Coding of quantization matrices using parametric models

ABSTRACT

Quantization matrix can be used to adjust quantization of transform coefficients at different frequencies. In one embodiment, a single fixed parametric model, such as a polynomial is used to represent a quantization matrix. Modulation of bit cost and complexity is achieved by specifying only the n first polynomial coefficients, the remaining ones being implicitly set to zero or other default values. One form of the single fixed polynomial is a fully developed polynomial in (x,y), where x,y indicate the coordinates of a given coefficient in a quantization matrix, with terms ordered by increasing exponent. Since higher exponents are the last ones, reducing the number of polynomial coefficients reduces the degree of the polynomial, hence its complexity. The polynomial coefficients can be symmetrical in x and y, and thus reducing the number of polynomial coefficients that need to be signaled in the bitstream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.17/267,014, filed Feb. 8, 2021, which is a National Phase entry under 35U.S.C. § 371 of International Application No. PCT/US2019/045687, filedAug. 8, 2019, which claims the benefit of European Patent ApplicationNo. 18306212.4, filed Sep. 17, 2018 and European Patent Application No.18306135.7 filed Aug. 23, 2018, the entirety of which is incorporated byreference herein.

TECHNICAL FIELD

The present embodiments generally relate to a method and an apparatusfor video encoding or decoding, and more particularly, to a method andan apparatus for coding quantization matrices in video encoding ordecoding.

BACKGROUND

To achieve high compression efficiency, image and video coding schemesusually employ prediction and transform to leverage spatial and temporalredundancy in the video content. Generally, intra or inter prediction isused to exploit the intra or inter frame correlation, then thedifferences between the original block and the predicted block, oftendenoted as prediction errors or prediction residuals, are transformed,quantized, and entropy coded. To reconstruct the video, the compresseddata are decoded by inverse processes corresponding to the entropycoding, quantization, transform, and prediction.

SUMMARY

According to an embodiment, a method for video decoding is presented,comprising: accessing a parametric model that is based on a sequence ofparameters; determining a plurality of parameters that correspond to asubset of said sequence of parameters; associating each parameter ofsaid plurality of parameters with a corresponding parameter of saidsubset of said sequence of parameters, to represent a quantizationmatrix; de-quantizing transform coefficients of a block of an imagebased on said quantization matrix; and reconstructing said block of saidimage responsive to said de-quantized transform coefficients.

According to another embodiment, a method for video encoding ispresented, comprising: accessing a parametric model that is based on asequence of parameters; determining a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associating eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; quantizing transform coefficients of a block of animage based on said quantization matrix; and entropy encoding quantizedtransform coefficients.

According to another embodiment, an apparatus for video decoding ispresented, comprising: means for accessing a parametric model that isbased on a sequence of parameters; means for determining a plurality ofparameters that correspond to a subset of said sequence of parameters;means for associating each parameter of said plurality of parameterswith a corresponding parameter of said subset of said sequence ofparameters, to represent a quantization matrix; means for de-quantizingtransform coefficients of a block of an image based on said quantizationmatrix; and means for reconstructing said block of said image responsiveto said de-quantized transform coefficients.

According to another embodiment, an apparatus for video encoding ispresented, comprising: means for accessing a parametric model that isbased on a sequence of parameters; means for determining a plurality ofparameters that correspond to a subset of said sequence of parameters;means for associating each parameter of said plurality of parameterswith a corresponding parameter of said subset of said sequence ofparameters, to represent a quantization matrix; means for quantizingtransform coefficients of a block of an image based on said quantizationmatrix; and means for entropy encoding quantized transform coefficients.

According to another embodiment, an apparatus for video decoding ispresented, comprising one or more processors, wherein said one or moreprocessors are configured to: access a parametric model that is based ona sequence of parameters; determine a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associate eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; de-quantize transform coefficients of a block of animage based on said quantization matrix; and reconstruct said block ofsaid image responsive to said de-quantized transform coefficients. Theapparatus can further comprise one or more memories coupled to said oneor more processors.

According to another embodiment, an apparatus for video encoding ispresented, comprising one or more processors, wherein said one or moreprocessors are configured to: access a parametric model that is based ona sequence of parameters; determine a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associate eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; quantize transform coefficients of a block of animage based on said quantization matrix; and entropy encode quantizedtransform coefficients. The apparatus can further comprise one or morememories coupled to said one or more processors.

According to another embodiment, a signal is formatted to include: avalue indicating a number of parameters in a plurality of parameters;said plurality of parameters, wherein each parameter of said pluralityof parameters is associated with a corresponding parameter of a subsetof a sequence of parameters, to represent a quantization matrix, whereina parametric model is based on said sequence of parameters; andtransform coefficients of a block of an image quantized based on saidquantization matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an embodiment of a video encoder.

FIG. 2 illustrates a block diagram of an embodiment of a video decoder.

FIG. 3 is a pictorial example illustrating 2D separable transform.

FIG. 4 illustrates the default intra and inter 8×8 quantization matrices(QMs) defined in HEVC.

FIG. 5 illustrates a process for determining the QM in HEVC.

FIG. 6 illustrates the up-right diagonal scanning and the default intra8×8 QM in HEVC.

FIG. 7 illustrates different transforms used in JEM (Joint ExplorationModel).

FIG. 8 illustrates the example matrices found in standards and encoders.

FIG. 9 illustrates two complex-shaped matrices.

FIG. 10 illustrates the AVC-type (left) and HVS (right) matrices forsize 8, 16 and 32.

FIG. 11A illustrates example matrix layers for an 8×8 size, and FIG. 11Billustrates 4 regions.

FIG. 12 illustrates fitting performance of [JCTVC-H0460] quadraticmodels for symmetric (left) or asymmetric (right) test intra matrices ofsize 8×8.

FIG. 13 illustrates an example of quantization matrix usingapproximation.

FIG. 14 illustrates 16×16 AV1 quantization matrices.

FIG. 15 illustrates the quantization matrix coding performance.

FIG. 16 illustrates a system for decoding the quantization matrix,according to an embodiment.

FIG. 17 illustrates a method for decoding the quantization matrix,according to an embodiment.

FIG. 18 illustrates a method for normalizing the matrix coordinates,according to an embodiment.

FIG. 19 illustrates the dynamic range of coefficients.

FIG. 20 illustrates the impact of coefficient accuracy on fit quality.

FIG. 21 illustrates the impact of coefficient bounds on fit quality.

FIG. 22 illustrates a quantization matrix generator with integercomputing, according to an embodiment.

FIG. 23 illustrates a method for generating a quantization matrix withinteger computing, according to an embodiment.

FIG. 24 illustrates the default HEVC intra matrix with 10-parameter fit.

FIG. 25 illustrates the impact of clipping on fitting errors.

FIG. 26 illustrates the default HEVC intra matrix with 6-parameter fit.

FIG. 27 illustrates a method for generating the QM in an encoder,according to an embodiment.

FIG. 28 illustrates a method for generating the QM in a decoder,according to an embodiment.

FIG. 29 illustrates a block diagram of a system within which aspects ofthe present embodiments can be implemented.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary video encoder 100, such as a HighEfficiency Video Coding (HEVC) encoder. FIG. 1 may also illustrate anencoder in which improvements are made to the HEVC standard or anencoder employing technologies similar to HEVC, such as a VVC (VersatileVideo Coding) encoder under development by JVET (Joint Video ExplorationTeam).

In the present application, the terms “reconstructed” and “decoded” maybe used interchangeably, the terms “encoded” or “coded” may be usedinterchangeably, and the terms “image,” “picture” and “frame” may beused interchangeably. Usually, but not necessarily, the term“reconstructed” is used at the encoder side while “decoded” is used atthe decoder side.

Before being encoded, the video sequence may go through pre-encodingprocessing (101), for example, applying a color transform to the inputcolor picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), orperforming a remapping of the input picture components in order to get asignal distribution more resilient to compression (for instance using ahistogram equalization of one of the color components). Metadata can beassociated with the pre-processing, and attached to the bitstream.

In HEVC, to encode a video sequence with one or more pictures, a pictureis partitioned (102) into one or more slices where each slice caninclude one or more slice segments. A slice segment is organized intocoding units, prediction units, and transform units. The HEVCspecification distinguishes between “blocks” and “units,” where a“block” addresses a specific area in a sample array (e.g., luma, Y), andthe “unit” includes the collocated blocks of all encoded colorcomponents (Y, Cb, Cr, or monochrome), syntax elements, and predictiondata that are associated with the blocks (e.g., motion vectors).

For coding, a picture is partitioned into coding tree blocks (CTB) ofsquare shape with a configurable size, and a consecutive set of codingtree blocks is grouped into a slice. A Coding Tree Unit (CTU) containsthe CTBs of the encoded color components. A CTB is the root of aquadtree partitioning into Coding Blocks (CB), and a Coding Block may bepartitioned into one or more Prediction Blocks (PB) and forms the rootof a quadtree partitioning into Transform Blocks (TBs). Corresponding tothe Coding Block, Prediction Block, and Transform Block, a Coding Unit(CU) includes the Prediction Units (PUs) and the tree-structured set ofTransform Units (TUs), a PU includes the prediction information for allcolor components, and a TU includes residual coding syntax structure foreach color component. The size of a CB, PB, and TB of the luma componentapplies to the corresponding CU, PU, and TU. In the present application,the term “block” can be used to refer, for example, to any of CTU, CU,PU, TU, CB, PB, and TB. In addition, the “block” can also be used torefer to a macroblock and a partition as specified in H.264/AVC or othervideo coding standards, and more generally to refer to an array of dataof various sizes.

In the exemplary encoder 100, a picture is encoded by the encoderelements as described below. The picture to be encoded is processed inunits of CUs. Each CU is encoded using either an intra or inter mode.When a CU is encoded in an intra mode, it performs intra prediction(160). In an inter mode, motion estimation (175) and compensation (170)are performed. The encoder decides (105) which one of the intra mode orinter mode to use for encoding the CU, and indicates the intra/interdecision by a prediction mode flag. Prediction residuals are calculatedby subtracting (110) the predicted block from the original image block.

The prediction residuals are then transformed (125) and quantized (130).The quantized transform coefficients, as well as motion vectors andother syntax elements, are entropy coded (145) to output a bitstream.The encoder may also skip the transform and apply quantization directlyto the non-transformed residual signal on a 4×4 TU basis. The encodermay also bypass both transform and quantization, i.e., the residual iscoded directly without the application of the transform or quantizationprocess. In direct PCM coding, no prediction is applied and the codingunit samples are directly coded into the bitstream.

The encoder decodes an encoded block to provide a reference for furtherpredictions. The quantized transform coefficients are de-quantized (140)and inverse transformed (150) to decode prediction residuals. Combining(155) the decoded prediction residuals and the predicted block, an imageblock is reconstructed. In-loop filters (165) are applied to thereconstructed picture, for example, to perform deblocking/SAO (SampleAdaptive Offset) filtering to reduce encoding artifacts. The filteredimage is stored at a reference picture buffer (180).

FIG. 2 illustrates a block diagram of an exemplary video decoder 200,such as an HEVC decoder. In the exemplary decoder 200, a bitstream isdecoded by the decoder elements as described below. Video decoder 200generally performs a decoding pass reciprocal to the encoding pass asdescribed in FIG. 1 , which performs video decoding as part of encodingvideo data. FIG. 2 may also illustrate a decoder in which improvementsare made to the HEVC standard or a decoder employing technologiessimilar to HEVC, such as a VVC decoder.

In particular, the input of the decoder includes a video bitstream,which may be generated by video encoder 100. The bitstream is firstentropy decoded (230) to obtain transform coefficients, motion vectors,picture partitioning information, and other coded information. Thepicture partitioning information indicates the size of the CTUs, and amanner a CTU is split into CUs, and possibly into PUs when applicable.The decoder may therefore divide (235) the picture into CTUs, and eachCTU into CUs, according to the decoded picture partitioning information.The transform coefficients are de-quantized (240) and inversetransformed (250) to decode the prediction residuals.

Combining (255) the decoded prediction residuals and the predictedblock, an image block is reconstructed. The predicted block may beobtained (270) from intra prediction (260) or motion-compensatedprediction (i.e., inter prediction) (275). In-loop filters (265) areapplied to the reconstructed image. The filtered image is stored at areference picture buffer (280). The decoded picture can further gothrough post-decoding processing (285), for example, an inverse colortransform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverseremapping performing the inverse of the remapping process performed inthe pre-encoding processing (101). The post-decoding processing may usemetadata derived in the pre-encoding processing and signaled in thebitstream.

As described above, the prediction residuals are transformed andquantized. Considering an M×N (M columns×N rows) residual block([U]_(M×N)) that is input to a 2D M×N forward transform, the 2Dtransform is typically implemented by applying an N-point transform toeach column (i.e., vertical transform) and an M-point transform to eachrow (i.e., horizontal transform) separately, as illustrated in FIG. 3 .Mathematically, the forward transform can be expressed as:

[C] _(M×N) =[A] ^(T) _(N×N) x[U] _(M×N) x[B] _(M×M)

where [A]_(N×N) is the N-point transform matrix applied vertically, and[B]_(M×M) the M-point transform matrix applied horizontally, and “T”(superscript) is the matrix transposition operator. Thus, the separabletransform consists in applying the horizontal and vertical transformssuccessively on each row and each column of the 2D residual block.

In HEVC, the 2D transforms use the same separable transform for bothhorizontal and vertical filtering. In other words, matrix B is same asmatrix A:

[C] _(M×N) =[A] ^(T) _(N×N) ×[U] _(M×N) ×[A] _(M×M)

Note that in HEVC, only square transforms are supported, which meansN=M.

The resulting M×N transform coefficients ([C]_(M×N)) are then subject toquantization to obtain quantized transform coefficients ([CQ]_(M×N)). InHEVC, Uniform Reconstruction Quantization (URQ) is used forquantization, which is conceptually equivalent to division byquantization step size (Qstep). More generally for video encoding, itshould be noted that other quantization methods may be used, forexample, by further considering a quantization rounding offset or usingnon-uniform quantization.

To quantize the transform coefficients, in HEVC, the encoder canspecify, for each transform block size and separately for intra andinter prediction, a customized quantization (scaling) matrix (QM) foruse in inverse-quantization scaling by the decoder. The quantizationmatrix is typically designed to provide more accuracy to coefficientscorresponding to frequencies more sensitive to human perception. Thegoal is to adapt the quantization of the transform coefficients toimprove the perceptual quality of the coded video, typically by takinginto account the properties of the human visual system to differentspatial frequencies of the signal. For example, low frequencies are lessquantized than the high frequencies.

Using the quantization matrix generally does not improve objectivefidelity as measured by mean-squared error (or, equivalently, PSNR), butit usually improves subjective fidelity. Default values for thequantization scaling matrices are specified in the standard, and theencoder can also choose to use customized values by sending arepresentation of those values at the sequence or picture level.

More specifically, the transform coefficients are quantized according tothe scaling values provided in quantization matrices. After thetransform is applied to the residual values, luma and chroma transformcoefficients in a Transform Block are individually quantized accordingto the integer scaling values that are present in the intra and interQMs for the specific component and block size: 6 matrices for each oftransform sizes 4×4, 8×8, 16×16, and 2 matrices for 32×32 (chroma in4:4:4 sampling mode reuses the matrices for 16×16), which makes a totalof 20 matrices. To reduce memory storage needs, matrix definition islimited to 8×8 resolution: full matrices for transform sizes 16×16 and32×32 are obtained by 2×2 and 4×4 sample repetition, respectively.

The HEVC specification defines default intra and inter QMs in HEVC, thatare based on human visual model. If the default matrices are not used asindicated using flags, ad hoc matrices explicitly signaled in thebitstream can be used.

The elements in the QMs apply different quantization scaling totransform coefficients at different frequencies in a Transform Block.Therefore, these QMs possess the capacity to control the quantizationstep size, and thus to adjust the quantization according to thefrequency of the transform coefficient. A Transform Block contains DCand AC transform coefficients, where the DC transform coefficient is thelowest frequency component and the AC coefficients correspond tofrequency components from low, medium to high. Because low frequencytransform coefficients are usually more important for video signalreconstruction, the default QMs in HEVC apply coarser quantization tomedium and high frequency AC transform coefficients.

The quantization at the encoder side is not normative and can be done indifferent ways. Nevertheless, it typically reproduces or approximatesthe following formula for an HEVC encoder:

CQ[x][y]=sign(C[x][y])*(abs(C[x][y])*invScale[QP%6]/QM[x][y]+quantOffset)>>quantShift

where

-   -   CQ is the block of quantized coefficients,    -   C is the block of transform coefficients,    -   QM is the quantization matrix,    -   invScale[k]={26214, 23302, 20560, 18396, 16384, 14564} with k=0,        . . . , 5,    -   QP is the quantization parameter,    -   quantShift is a parameter that depends on the scaling involved        by the forward transform applied at the encoder, on the        transform block size, on the signal bit depth, and on QP.        -   In the HEVC reference encoder,            quantShift=14+QP+TransformShift, where TransformShift            relates to the scaling involved by the forward transform,    -   quantOffset is a parameter that depends on parameter quantShift,        for instance, quantOffset=1<<(quantShift—1),    -   (x,y) is the location of the coefficient,    -   abs(r) is the absolute value of r,    -   sign(r)=−1 if r<0, 1 otherwise,    -   “*” is the scalar multiplication, “/” is the integer division        with truncation of the result toward zero, and “%” is the        modulus operator.

At the decoder side, the QM can be applied in the de-quantizationconforming to the HEVC specification based on the following formula:

C′[x][y]=(CQ[x][y]*QM[x][y]*levScale[QP%6]<<(QP/6))+(1<<(bdShift−1)))>>bdShift

where

-   -   CQ is the block of quantized coefficients,    -   C′ is the block of de-quantized coefficients,    -   QM is the quantization matrix,    -   levScale[k]={40, 45, 51, 57, 64, 72} with k=0, . . . , 5,    -   QP is the quantization parameter,    -   bdShift is defined as follows for the HEVC usual profiles:        bdShift=Max(20—bitDepth, 0), bitDepth is the bit depth of the        samples of the considered component (e.g., Y, Cb or Cr),    -   (x,y) is the location of the coefficient.

Default QMs are defined in HEVC for 8×8 transform blocks only, in orderto reduce the memory storage needs. The default intra and inter 8×8 QMsdefined in HEVC are shown below and illustrated in FIG. 4 (note thatthey are both symmetric). The QMs for larger blocks are obtained byupsampling the 8×8 QMs. To create a 16×16 QM, each entry in an 8×8 QM isreplicated into a 2×2 region. To create a 32×32 QM, each entry in an 8×8QM is replicated into a 4×4 region. For 4×4 transform blocks, thedefault QM is flat (all components are equal to 16). The default QMsonly depend on the intra/inter mode, but are the same for the Y, Cb andCr components. The replication rule could also be applied to rectangularblocks (e.g., rectangular blocks in JEM).

$\begin{pmatrix}16 & 16 & 16 & 16 & 17 & 18 & 21 & 24 \\16 & 16 & 16 & 16 & 17 & 19 & 22 & 25 \\16 & 16 & 17 & 18 & 20 & 22 & 25 & 29 \\16 & 16 & 18 & 21 & 24 & 27 & 31 & 36 \\17 & 17 & 20 & 24 & 30 & 35 & 41 & 47 \\18 & 19 & 22 & 27 & 35 & 44 & 54 & 65 \\21 & 22 & 25 & 31 & 41 & 54 & 70 & 88 \\24 & 25 & 29 & 36 & 47 & 65 & 88 & 115\end{pmatrix}\begin{pmatrix}16 & 16 & 16 & 16 & 17 & 18 & 20 & 24 \\16 & 16 & 16 & 17 & 18 & 20 & 24 & 25 \\16 & 16 & 17 & 18 & 20 & 24 & 25 & 28 \\16 & 17 & 18 & 20 & 24 & 25 & 28 & 33 \\17 & 18 & 20 & 24 & 25 & 28 & 33 & 41 \\18 & 20 & 24 & 25 & 28 & 33 & 41 & 54 \\20 & 24 & 25 & 28 & 33 & 41 & 54 & 71 \\24 & 25 & 28 & 33 & 41 & 54 & 71 & 91\end{pmatrix}$

The intra default QM is based on the human visual system, as explainedin a patent application by M. Hague et al. (U.S. patent application Ser.No. 13/597,131, Publication No. US2013/0188691, “Quantization matrixdesign for HEVC standard”, hereinafter [US20130188691]) or in an articleby Long-Wen Chang et al., entitled “Designing JPEG quantization tablesbased on human visual system,” Signal Processing: Image Communication,Volume 16, Issue 5, pp 501-506, January 2001 (hereinafter [Chang]), andcan be derived as explained in the following process. At first, f, theradial frequency in cycles per degree of the visual angle correspondingto the coefficient at location (x,y), is defined as:

$\begin{matrix}{{{f\left( {x,y} \right)} = {\frac{K}{{\Delta \cdot 2}N}\frac{\sqrt{x^{2} + y^{2}}}{S(\theta)}{where}}}{{\theta = {{arc}{\tan\left( \frac{u}{v} \right)}}},{{S(\theta)} = {{\frac{1 - s}{2}*{\cos\left( {4\theta} \right)}} + \frac{1 + s}{2}}},}} & (1)\end{matrix}$

and N is the block width or height (u=0, . . . , N−1, v=0, . . . , N−1,typically N=8), Δ, K and s are constant parameters (to get the HEVC 8×8QM, A should be set to 0.00025, K to 0.00893588, s to 0.7).

Then the Modulation Transfer Function H(f) is defined as

$\begin{matrix}{{H(f)} = \left\{ \begin{matrix}{2.2 \cdot \left( {0.192 + {0.114 \cdot f}} \right) \cdot {\exp\left( {- \left( {0.114 \cdot f} \right)^{1.1}} \right)}} & {{{if}f} > f_{\max}} \\1 & {otherwise}\end{matrix} \right.} & (2)\end{matrix}$

where f_(max)=8 (cycles per degree). The QM values are computed asRound(16/H(f)) where Round(x) gives the nearest integer value to x.

The inter default QM can be derived from the intra default QM using thefollowing process:

-   -   1. QMinter(0,0)=QMintra(0,0)    -   2. For n=1 . . . N−1        -   QMinter(0,n)=QMinter(0,n−1)+int(S1*(QMintra(0,n)−QMintra(0,n−1))+0.5),        -   With S1=0.714285714    -   3. QMinter(0,N−1)=QMintra(0,N−1)    -   4. For m=1 . . . N−1        -   QMinter(m,N−1)=QMinter(m−1,N−1)+int(S3*(QMintra(m,N−1)−QMintra(m−1,N−1))+0.5),        -   With S3=0.733333333, where int(r) is the nearest integer            value of r,    -   5. For m=1 . . . N−1, For n=0 . . . N−2        -   QMinter(m, n)=QMinter(m−1, n+1)

The coefficients of the quantization matrices are signaled in thebitstream using scaling lists. There is one scaling list per block size,indexed by the parameter sizeld, and per mode, indexed by the parametermatrixId, both specified as follows:

-   -   sizeId=0—block 4×4, 1—block 8×8, 2—block 16×16, 3—block 32×32    -   matrixId=0—intra Y, 1—intra U, 2—intra V, 3—inter Y, 4—inter U,        5—inter V For 32×32 blocks, matrixId can only be set to 0 or 3.

The syntax in HEVC specification for signaling the scaling list isreproduced as follows.

scaling_list_data( ) { Descriptor  for( sizeId = 0; sizeId < 4; sizeId++)   for( matrixId = 0; matrixId < 6; matrixId += ( sizeId = = 3) ? 3: 1) {    scaling_list_pred_mode_flag[ sizeId ][ matrixId ] u(1)    if(!scaling_list_pred_mode_flag_sizeId ][ matrixId ] )    scaling_list_pred_matrix_id_delta[ sizeId ][ matrixId ] ue(v)   else {     nextCoef = 8     coefNum = Min( 64, (1 << (4 + ( sizeId <<1 ) ) ) )     if( sizeId > l) {      scaling_list_dc_coef_minus8[sizeId - 2 ][ matrixId ] se(v)      nextCoef                    =scaling_list_dc_coef_minus8[ sizeId − 2 ][ matrixId ] + 8     }     for(i = 0; i < coefNum; i++) {      scaling_list_delta_coef se(v)     nextCoef = ( nextCoef + scaling_list_delta_coef + 256 ) % 256     ScalingList[ sizeId ][ matrixId ][ i ] = nextCoef     }    }   } }

For each possible value of the two parameters sizeld and matrixId,process 500 illustrated in FIG. 5 is applied to determine the actual QMfrom either the default QM, or from explicit decoded values for anexemplary HEVC decoder. At step 510, the decoder checks whether thesyntax element scaling_list_pred_mode_flag[sizeId][matrixId] is equal to0. If not, at step 550, the decoder decodes scaling list values from thevariable ScalingList[sizeId][matrixId][i], i=0, . . . , coefNum−1, andbuilds the QM by the up-right diagonal scanning order as shown in theleft of FIG. 6 . If scaling_list_pred_mode_flag[sizeId][matrixId] isequal to 0, at step 520, the decoder checks whether the syntax elementscaling_list_pred_matrix_id_delta[sizeId][matrixId] is equal to 0. Ifnot, matrixId is modified (530) based onscaling_list_pred_matrix_id_delta[sizeId][matrixId]. At step 540, thescaling list values from the default scaling list values identified bysizeld and matrixId are obtained, and the QM is built, where the defaultmatrix for 8×8 intra is shown in the right of FIG. 6 .

As described above, previous video codecs, such as those conforming toHEVC, were based on 2D separable transforms using the same vertical andhorizontal transforms. Therefore, the derived QMs were 2D QMs, ingeneral symmetric, adapted to these 2D transforms.

However, in the current JEM, five different horizontal/verticaltransforms are defined, derived from five transforms as shown in Table 1and illustrated for 4×4 size in FIG. 7 . Flags are used at the CU level,for size from 4×4 to 64×64, to control the combination of transforms.When the CU flag is equal to 0, DCT-II is applied as horizontal andvertical transform. When the CU flag is equal to 1, two additionalsyntax elements are signalled to identify the horizontal and verticaltransforms to be used. Note that other horizontal/vertical transformscould also be considered, such as the identity transform (whichcorresponds to skipping the transform in one direction).

TABLE 1 Transform basis functions of DCT-II/V/VIII and DST-I/VII forN-point input in JEM. Transform Type Basis function T_(i)(j), i, j = 0,1, . . . , N − 1 DCT-II${{T_{i}(j)} = {\omega_{0} \cdot \sqrt{\frac{2}{N}} \cdot {\cos\left( \frac{\pi \cdot i \cdot \left( {{2j} + 1} \right)}{2N} \right)}}},$${{where}\omega_{0}} = \left\{ \begin{matrix}\sqrt{\frac{2}{N}} & {i = 0} \\1 & {i \neq 0}\end{matrix} \right.$ DCT-V${{T_{i}(j)} = {\omega_{0} \cdot \omega_{1} \cdot \sqrt{\frac{2}{{2N} - 1}} \cdot {\cos\left( \frac{2{\pi \cdot i \cdot j}}{{2N} - 1} \right)}}},$${{where}\omega_{0}} = \left\{ {\begin{matrix}\sqrt{\frac{2}{N}} & {i = 0} \\1 & {i \neq 0}\end{matrix},{\omega_{1} = \left\{ \begin{matrix}\sqrt{\frac{2}{N}} & {j = 0} \\1 & {j \neq 0}\end{matrix} \right.}} \right.$ DCT-VIII${T_{i}(j)} = {\sqrt{\frac{4}{{2N} + 1}} \cdot {\cos\left( \frac{\pi \cdot \left( {{2i} + 1} \right) \cdot \left( {{2j} + 1} \right)}{{4N} + 2} \right)}}$DST-I${T_{i}(j)} = {\sqrt{\frac{2}{N + 1}} \cdot {\sin\left( \frac{\pi \cdot \left( {i + 1} \right) \cdot \left( {j + 1} \right)}{N + 1} \right)}}$DST-VII${T_{i}(j)} = {\sqrt{\frac{4}{{2N} + 1}} \cdot {\sin\left( \frac{\pi \cdot \left( {{2i} + 1} \right) \cdot \left( {j + 1} \right)}{{2N} + 1} \right)}}$

For the intra case, the set of possible transforms depends on the intramode. Three sets are defined as follows:

-   -   Set 0: DST-VII, DCT-VIII    -   Set 1: DST-VII, DST-I    -   Set 2: DST-VII, DCT-V        For each intra mode and each transform direction        (horizontal/vertical), one of these three sets is enabled. For        the inter case, only DST-VII and DCT-VIII are enabled, and the        same transform is applied for both horizontal and vertical        transforms.

As described above, in many codecs, quantization matrices are used. Forexample, FIG. 8 shows six example matrices found in standards andencoders. From left to right, top to bottom, the matrices are from JPEG,MPEG2-intra, H264-intra, HEVC-intra, an actual DTT H.264 encoder andsatellite HEVC broadcast stream. However, as many encoders do not usethe default matrices specified in the codec specification, they need tobe explicitly coded. The cost of coding such quantization matrices maybecome significant.

More specifically, one or more of the following problems may arise:

-   -   There may be many quantization matrices: the matrices may depend        on the horizontal/vertical size of the transform block, the        horizontal/vertical transform type, and the coding mode of the        Coding Unit containing the transform block; the number of        possible combinations may become quite large if considering 4,        8, 16, 32, 64-sized transforms with all rectangular variants,        intra/inter, and Y/U/V, this makes 150 matrices (to be compared        with 8 for H.264 and 20 for HEVC).    -   Recent codecs tend to use larger-sized transforms, which in turn        require transmitting larger-sized quantization matrices, which        means a greater number of coefficients.    -   As indicated above, one can choose alternate matrices not        matching the default ones found in the specification, for        psychovisual improvements or other reasons.    -   The quantization matrix may be asymmetric, for instance to adapt        to interlaced content, or anamorphic content.    -   It may be required to change the quantization matrices per        picture, in order to better adjust the quantization parameter        (for instance to emulate QP larger than 51, which is the current        maximum value specified in AVC and in HEVC, or to emulate        fractional QP).    -   It may be required for the bitrate regulation, to sacrifice some        frequencies; this typically appears in “panic mode”, when the        buffer is close to be full; in this case, it is penalizing to        spend many bits to code the quantization matrices.    -   It may be required to adjust the transform coefficients        differently depending on the temporal distance of the temporal        frame used for predicting the block (the residual statistics        strongly depend on this temporal distance), thus requiring        frequent change of quantization matrices.    -   Even a simple change can be costly if transmission of the full        matrix is required.    -   In current codecs (e.g., H.264 and HEVC), the coding cost of the        quantization matrices is somewhat related to their complexity        thanks to DPCM coding, but this could be pushed further.        Similarly, if prediction is used, coding cost should be related        to the complexity of change compared to the predictor.

Another aspect addressed here is that in current codecs, quantizationmatrix is used as a multiplier in the dequantization process, which addscomplexity compared to no-matrix: it needs two multiplications insteadof one and increases the dynamic range of intermediate results. Also,the HVS model underlying the default HEVC matrices follows anexponential trend, which is difficult to match with a low complexitymodel and few parameters.

Some problems have already been studied in JCT-VC, during thedevelopment of HEVC. For example, increase of transform sizes and typeshave raised discussions and proposals around quantization matricescoding efficiency:

-   -   Evidence of problem has been provided in an article by K.        Sato, H. Sakurai, entitled “Necessity of Quantization Matrices        Compression in HEVC,” JCTVC-E056, JCT-VC 5th Meeting: Geneva, C        H, March 16-23, 2011 (hereinafter [JCTVC-E056]).    -   An article by M. Zhou, V. Sze, entitled “Compact representation        of quantization matrices for HEVC,” JCTVC-D024, JCT-VC 4th        Meeting: Daegu, Korea, Jan. 20-28, 2011 (hereinafter        [JCTVC-D024]), proposed a coding method involving symmetries,        subsampling, and linear interpolation.    -   An article by J. Tanaka, Y. Morigami, and T. Suzuki, entitled        “Quantization Matrix for HEVC,” JCTVC-E073, JCT-VC 5th Meeting:        Geneva, C H, Mar. 16-23, 2011 (hereinafter [JCTVC-E073]),        proposed a coding method with optional prediction from linear        models, prediction from other matrices, symmetries, and        non-uniform residue quantization.    -   An article by G. Korodi and D. He, entitled “QuYK: A Universal,        Lossless Compression Method for Quantization Matrices,”        JCTVC-E435, JCT-VC 5th Meeting: Geneva, Mar. 16-23, 2011        (hereinafter [JCTVC-E435]), proposed two methods, one with        advanced entropy coding, and another with symmetries,        per-diagonal affine or quadratic prediction, and zero-tree        coding of residual.    -   An article by E. Maani, M. Hague, A. Tabatabai, entitled        “Parameterization of Default Quantization Matrices,” JCTVC-G352,        JCT-VC 7th Meeting: Geneva, C H, Nov. 21-30, 2011 (hereinafter        [JCTVC-G352]), proposed a quadratic model.    -   An article by Y. Wang, J. Zheng, X. Zheng, Yun He, entitled        “Layered quantization matrices compression,” JCTVC-G530, JCT-VC        7th Meeting: Geneva, Nov. 21-30, 2011 (hereinafter        [JCTVC-G530]), proposed a coding method with a hierarchical        iterative refinement.    -   An article by R. Joshi, J. S. Rojals, M. Karczewicz, entitled        “Compression and signaling of quantizer matrices,” JCTVC-G578,        JCT-VC 7th Meeting: Geneva, C H, Nov. 21-30, 2011 (hereinafter        [JCTVC-G578]), proposed a raster scan coding mode (instead of        diagonal) with modified differential coding, on top of symmetry        rules.    -   An article by M. Hague, E. Maani, A. Tabatabai, entitled        “High-level Syntaxes for the Scaling List Matrices Parameters        and Parametric coding,” JCTVC-H0460, JCT-VC 8th Meeting: San        José, Calif., USA, Feb. 1-10, 2012 (hereinafter [JCTVC-H0460]),        proposed a prediction using 3 parametric models (quadratic or        HVS).    -   An article by S. Jeong, Hendry, B. Jeon, J. Kim, entitled        “HVS-based Generalized Quantization Matrices,” JCTVC-I0518,        JCT-VC 9th Meeting: Geneva, C H, Apr. 27 — May 7, 2012        (hereinafter [JCTVC-I0518]) proposed an HVS-based model with a        single parameter.    -   An article by R. Joshi, J. S. Rojals, M. Karczewicz, entitled        “Quantization matrix entries as QP offsets,” JCTVC-I0284, JCT-VC        9th Meeting: Geneva, C H, Apr. 27 — May 7, 2012 (hereinafter        [JCTVC-I0284]), suggested the use of quantization matrices as QP        offsets.

Evidence of increase of quantization matrix coding cost has been shownin [JCTVC-E056]. This has encouraged the development of techniques formore efficient coding of quantization matrices. An article by J. Tanaka,Y. Morigami, and T. Suzuki, entitled “Enhancement of quantization matrixcoding for HEVC”, JCT-VC 6th Meeting: Torino, IT, Jul. 14-22, 2011(hereinafter [JCTVC-F475]), provided complex-shaped matrices, as shownin FIG. 9 , to test the coding efficiency of the various proposals. Thequantization matrices provided in FIG. 9 are both for 16×16, where theleft one is symmetric and the right one is asymmetric. They are notmeant to be meaningful, but designed for stress test: hard to encode,but not too much.

JCTVC-D0241 involves both x/y and central point symmetries, plussubsampled representation and linear interpolation reconstruction. Notethat x/y symmetry means matrix symmetry, namely, M(x,y) =M(y,x). x/ysymmetry is called “135 degree symmetry” in [JCTVC-D024]. Central pointsymmetry refers to “45 degree symmetry” in [JCTVC-D024], andmathematically, M(x,y)+M(N−1−y, N−1−x)=C, where N is the size of the(square) matrix, C is a constant, and x, y=0, 1, 2, . . . N−1.[JCTVC-D024] also proposed restriction to positive-only DPCM coding, butthat was withdrawn afterwards.

Central-point symmetry is usually not relevant, and x/y symmetry is notalways true. One benefit of the [JCTVC-D024] method comes fromsubsampled representation, that leads to about 2.5× reduction in bitcost with low error. However, since upsampling is not fully regular, itadds a little complexity to the specification.

JCTVC-E0731 offers several coding modes. At first, one mode involvedaffine prediction of horizontal, vertical, and diagonal axes, and linearinterpolation for the rest; but that mode was abandoned in the 2^(nd)version ([JCTVC-F475]). The rest involves prediction from other matrices(with fixed tree and upscale), non-uniform residue quantization(quantization matrix for quantization matrix), x/y symmetry, and variousscanning and coding options (raster DPCM, zigzag DPCM, or zigzag RLE),and VLC entropy coding. It suggested interpretation of quantizationmatrices as QP-offset in the first version.

In the [JCTVC-E073] method, the main reduction of bit cost comes fromquantization, which creates moderate errors. However, there are manyoptions, and the specification seems too complex for the purpose. The3-axes affine model followed by linear interpolation is interesting, butprobably not better than a single-stage polynomial model with the samenumber of parameters. Interpretation as a QP-offset has not been furtherdiscussed until [JCTVC-I0284] (see below).

JCTVC-E4351 proposed two methods: the first one is an advanced entropycoding with string substitution and arithmetic coding; the second onehas three modes: an asymmetry mode where each up-right diagonal ispredicted using a quadratic model with its own set of parameters, an x/ysymmetry mode with an affine model instead of a quadratic one, and anx/y+central-point symmetry mode where only half of the diagonals arecoded. Parameters for each diagonal are transmitted as indices to afinite set of coefficients, plus an offset. Residual is coded using azero-tree.

For [JCTVC-E435], the first method is a new entropy coding method andseems too complex for the purpose. In the second method, modeling eachdiagonal with coarse-grained low-degree polynomial is interesting, butprobably too complex compared to a global parametric model: diagonalsare clearly correlated in real life. The test matrix (same as[JCTVC-D024]) is a bit too simple and biased towards half-diagonalzigzag scanning efficiency; it is not clear whether compressionperformance comes from per-diagonal affine model or zero-tree coding.

JCTVC-G3521 and [US20130188691] proposed a quadratic polynomial modelfor default matrices. Matrix coefficients are approximated by

QM(x,y)=(a(x ² +y ²)+bxy+c(x+y)+d)/2^(q)

with x and y the position of the coefficient (from 0 to N-1 for an N×Nmatrix), (a, b, c, d) the parameters of the model as shown in Table 2,and q the bit precision which is typically 10. It is said that simplescaling of (a, b, c, d) would provide a straightforward extension forother matrix sizes: for a 2N×2N matrix, they are divided by (4, 4, 2, 1)respectively.

TABLE 2 Proposed parameters for AVC-type and HVS default matrices TypeSize a b c d AVC 8 × 8 −56 −127 3364 6898 intra 16 × 16 −14 −32 16826898 32 × 32 −4 −8 841 6898 HVS 8 × 8 619 1277 −4904 20249 intra 16 × 16171 369 −3039 23826 32 × 32 45 99 −1689 26059

This model provides a good fit for AVC-like matrices (easy since theynearly match a plane), but not so for HVS, and it is restricted toconvex shapes. Also, dynamic range of parameters vary greatly with x ory exponent, especially for large sizes; this would lead to some accuracyproblems with lower bit precision q. FIG. 10 illustrates the AVC-type(left) and HVS (right) matrices for size 8, 16 and 32, where the blackgrid shows the model and the gray surface shows the fitting target.

In [JCTVC-G352], reference is made to a work from Zhang et al. where aquadratic model is used internally by an encoder to optimizequantization matrices, with optimized matrices supposed to betransmitted in a traditional way (see, e.g., a patent application by H.Zhang et al., “Method and apparatus for modeling quantization matricesfor image/video encoding”, U.S. Pat. No. 8,326,068, hereinafter [U.S.Pat. No. 8,326,068]).

JCTVC-G5301 and an article by Y. Wang, J. Zheng, Yun He, entitled“Layered quantization matrices representation and compression,”JCTVC-H0314, JCT-VC 8th Meeting: San José, Calif., USA, Feb. 1-10, 2012(hereinafter [JCTVC-H314]), proposed a new method. It follows ahierarchical approach, where each position in the matrix is assigned alayer index (this assignment matrix is specific to each matrix size).FIG. 11A shows example matrix layers for an 8×8 size. First, thecoefficients at root level positions are decoded, then coefficients ofthe next layer are predicted using linear interpolation and correctedwith a residual; the process continues to the next layer, and so on.Residuals can be quantized with different scales for the 4 regionsdefined in [JCTVC-E073] (see FIG. 11B) and can be transmitted for alimited number of layers (e.g., only the first 2 layers), the remainingones being zero. There is an x/y symmetric mode where only half theresiduals are transmitted. Residuals are coded with DPCM or RLEdepending on the layer. Instead of being coding explicitly as justdescribed, matrices can also be copied from other ones from the samepicture (using a fixed tree, from greater size to lowest, using regularsubsampling), or coded differentially from the same matrix of theprevious picture.

This method outperforms some other methods, with an interestinghierarchical approach, but is typically too complex for the purpose.Specifically, linear interpolation with unevenly spaced anchors iscomplex, with ratios that can be numbers like 1/3 or 3/7.

JCTVC-G5781 proposed a simple coding method on top of symmetry rules andsubsampling: coefficients are scanned in raster order, each one ispredicted with the maximum of upper and left neighbors, and correctedwith a residual. Next one uses the corrected coefficient for prediction.Residuals are exp-golomb (EG) coded, with signed to unsigned mappingfavoring positive value, since matrix coefficients are typicallyincreasing. Residual transmission can be limited to a sub-block of thematrix (e.g., upper-left corner), the remaining one being inferred to O.The second version, as described in an article by R. Joshi, J. S.Rojals, M. Karczewicz, entitled “Coding of quantization matrices basedon modified prediction and mapping to unsigned values,” JCTVC-H0451,JCT-VC 8th Meeting: San José, Calif., USA, Feb. 1-10, 2012 (hereinafter[JCTVC-H0451]), added golomb-rice coding with variable order.

This method has good performance (about 40% less bits) in the asymmetriccase, with very low complexity. However this might be biased by the testmatrix, which is quasi monotonically increasing, thus well fit formax(left, top) prediction, and better than diagonal because ofsignificant asymmetry. Note that 15 of the 40% comes from offset insigned to unsigned mapping, thanks to increasing matrix values. Thegains are much lower for the symmetric case, and globally lower forsmaller sizes than for big sizes.

JCTVC-H04601 proposed a parametric model-based coding method. First, aparametric model is selected among three possible ones:

TABLE 3 Parametric models proposed in [JCTVC-H0460] Type ParametersDefinition Symmetric 4 QM(x, y) = a(x² + y²) + quadratic bxy + c(x +y) + d (same as [JCTVC-G352]) Asymmetric 6 According to the document:quadratic QM(x, y) = (a₁x + b₁y + c₁)(a₂x + b₂y + c₂) According to thecode: QM(x, y) = ax² + by² + cxy + dx + ey + f HVS (like default matrix)4 $\begin{matrix}{{H\left( {x,y} \right)} = {{a\left( {b + {c \cdot {f\left( {x,y} \right)}}} \right)} \cdot}} \\{\exp\left( {- \left( {c \cdot {f\left( {x,y} \right)}} \right)^{d}} \right)} \\{{{QM}\left( {x,y} \right)} = \frac{16}{H\left( {x,y} \right)}} \\{{with}{f\left( {x,y} \right)}{the}{frequency}{function}{in}} \\{\lbrack{Chang}\rbrack{or}{{Eq}.(1)}\left( {{not}{found}{in}{the}} \right.} \\\left. {{test}{code}} \right)\end{matrix}$

Then, optionally, a residual can be transmitted with a conventionalmethod (e.g., zigzag scan and symmetry rule). Test results are given forparameters coded with EG5 and normalized to 32, and residuals. Matrixsizes greater than 8×8 are generated using an 8×8 representationfollowed by upsampling.

The plurality of parametric models and other options (like residualcoding methods) makes the method complex. Moreover, the HVS model iscomputationally demanding. Results with residual do not show betterperformance than traditional method, but it is argued that if residualsare not needed (i.e., model is considered sufficient for the purpose),the representation is very compact. But the model is probably notsufficient since it is limited to 2^(nd)-degree polynomials, and the HVSversion does not provide more degrees of freedom. In FIG. 12 , fittingperformance of [JCTVC-H0460] quadratic models for symmetric (left) orasymmetric (right) test intra matrices of size 8×8 is shown, using SSEoptimization and positive constraint, where the gray surface representsthe target and the black grid the model. Results are in 38 and 56 bits(anchor is in 334 and 380).

JCTVC-I05181 proposed a generalized HVS model to transmit scalingmatrices with a single α parameter:

${Q{M\left( {x,y} \right)}} = \frac{16}{{H\left( {x,y} \right)}^{\alpha}}$

With H the same as in [Chang] or Eq. (2). A simplified computation isalso proposed, using an approximation of 1/H as powers of two. Anexample using the approximation is shown in FIG. 13 .

With only one parameter, this model is very compact, and the exponentprobably makes it well fit for intra to inter conversion and temporaldistance adaptation. However, the underlying formulas are quite complexand computationally intensive. The simplified computation techniqueintroduces coarse approximation that may not be acceptable (e.g., thelower triangle half is fully flat). Moreover, restriction ofquantization matrices to this class of matrices, with only one degree offreedom, is probably too severe.

JCTVC-I02841 suggested the use of quantization matrices as QP offsets(as already proposed in [JCTVC-E073]), explaining that it makes QPadjustment much more uniform in both directions, and also simplifies thedequantization process and potentially saves memory to in someimplementations. It requires a local increase of QP granularity, byaugmenting the resolution of levScale table in the dequantizationprocess. Matrix would be coded in offset-binary (0-119 with 64 asmid-point).

During the standardization process, the JCT-VC group was very reluctantto increase the complexity of matrix coding, and memory storage wasconsidered as critical as coding cost. Thus, matrix subsampling withrepetition was chosen because it reached both goals without additionalcomplexity and with minor quality impact, apart from that the design waskept mostly unchanged from AVC. Various attempts at non-uniformsubsampling were rejected, because of increased complexity and unclearbenefits. Using symmetry also increases complexity and was notconsidered worth the effort.

For large matrix sizes, sample repetition was preferred to linearinterpolation because the dequantization process can then use thereduced-size representation without needing any on-the-fly computation.It is noted that subjective tests conducted during JCT-VC meeting Hsuggested that small errors in matrix coefficients had no visual impact,as described in a break-out group report on quantization matricessubjective viewing by T. Suzuki, K. Sato, X. Zhang, R. Joshi, J. Zheng,M. Zhou, entitled “Report on Quantization Matrices Subjective Viewing,”JCTVC-H0730, JCT-VC 8th Meeting: San José, Calif., USA, Feb. 1-10, 2012(hereinafter [JCTVC-H0730]).

Since small errors seem acceptable, and quantization matrix data isusually smooth, a parametric model would be a good fit. In fact, thedata compression approach adopted to compare the different JCT-VCproposals may be misleading, since it is not focused on the real needsof quantization matrix designs. The idea of degrees of freedom may be abetter match. Also, anticipated increase of the number of quantizationmatrices in the next standard pushes towards a more compactrepresentation.

The emerging AV1 video coding standard also uses quantization matrices.Currently it makes use of a set of 16 possible matrix sets, hard-codedin both encoder and decoder. One matrix set comprises matrices for eachblock size, luma, and chroma. Like in HEVC, each matrix coefficient isused as a multiplicative factor of the quantization step for thematching coefficient of the transformed block.

An index specifying which set out of 16 to use can be signaled by theencoder at picture level or 64×64 block level, and separately for eachcolor component. This index is called “QM level” by AV1. FIG. 14 shows16×16 AV1 quantization matrices, for luma and chroma index 0 at the toprow, and for luma index 4 and 8 at the bottom row.

According to the hard-coded tables, increasing levels lead to flatterand flatter matrices. Matrices for index 15 are fully flat (withnormalization value=32). Actually, a decoder interprets index 15 as“bypass” and skips matrix in the dequantization process in that case.

Here, the problem of transmitting many matrices is completely avoided byrelying on hard-coded matrices. Flexibility is provided by choosingbetween 16 possibilities, but this is quite limited: this can becompared to varying the c factor in the HVS model of [JCTVC-H0460], orthe alpha exponent in [JCTVC-I0518]. However, compared to HEVC,flexibility is improved by the ability to select the matrix at the blocklevel.

On the other hand, hard-coded matrices take up much space in read-onlymemory, code, and specification. In particular, currently there are107008 8-bit values in 6767 lines of code for decoder matrices, and asmany for the encoder.

The present application proposes a compact representation ofquantization matrices using a simple fixed polynomial model, withmonomials sorted by increasing complexity, and a variable number ofparameters, so that trading more bits for better accuracy reduces tocode more or fewer parameters. In the following, some advantages of theproposed techniques are provided in comparison with other known works.

Advantages Prior art A single fixed model, leading to simple Some fixedmodels were proposed, but either implementation too restrictive or toocomplex Variable number of parameters, directly Parametric models withfixed number of driving coding cost, accuracy and parameters. Monomialsordered by complexity decreasing complexity. Very compact representation(typically Parametric models with comparable bit 10x less bits thantraditional approach for savings 8 × 8 matrix), saving bits in a videobitstream, and allowing a higher variety of quantization matrices, thusimproving psychovisual adaptation High flexibility, allowing complexParametric models with lacking flexibility, shapes or layered method butless bit-efficient, or full specification with more bits. Independent ofblock size Block-size dependent Interpretation as QP-offset can bringSome benefits can apply too additional benefits Computations can beperformed on the Applicable for simple parametric models fly in thedequantization process, (e.g., quadratic) minimizing memory needs

The proposed technique can be used as direct matrix coding, or as aresidual from a prediction, and either for default matrices or custom(transmitted) ones. In one embodiment, a residual using another codingmethod may be added to adjust QM coefficients further.

As described above, we anticipate a high pressure on bit cost ofquantization matrices in the future video standard. In variousembodiments, parametric models are used to offer a very compactrepresentation, at the expense of some loss of freedom.

Since quantization matrices are usually very smooth, full control overeach matrix coefficient is not required: this is clear from the resultsof subjective viewing in JCT-VC meeting H where no difference could bedetected between lossy and lossless QM transmission techniques, and thefinal decision to represent big matrices with an 8×8 resolution. Thus, asolution to encode the quantization matrix can be defining the globalshape with “enough” freedom.

In one embodiment, a polynomial model is proposed, because amongpossible parametric surface models, polynomials probably offer thehighest flexibility for the lowest complexity.

Note that even if the proposed representation is block-size independent,it is compatible with an expansion limited to 8×8 maximum, withrepetition for higher sizes, like in HEVC. FIG. 15 illustrates thequantization matrix coding performance, where the simplest variant ofthe model proposed here (using 3, 4, 6, 8, 10, 11, 13, 15, 17, 19, or 21parameters) is compared to the layered method proposed by [JCTVC-G530](best “lossy” method proposed for HEVC), with AVC/HEVC coding method asa reference, for some 8×8 matrices (comparable results are obtained forother matrices). The proposed model generally extends and outperforms[JCTVC-G530] on the low-bitrate side.

FIG. 16 illustrates system 1600 for decoding the quantization matrix,according to an embodiment. From the input bitstream, parameter decoder1610 obtains model parameters, e.g., polynomial coefficients {Pi}, forthe current quantization matrix. Then quantization matrix generator 1620generates the current quantization matrix based on the matrix size andthe model parameters.

FIG. 17 illustrates method 1700 for decoding the quantization matrix,according to an embodiment. Method 1700 starts at step 1705. From theinput bitstream, the decoder 1710 obtains model parameters, e.g.,polynomial coefficients {Pi}, for the current quantization matrix. Thenthe decoder generates 1720 the current quantization matrix based on thematrix size and the model parameters. Method 1700 ends at step 1799.Method 1700 can be implemented in system 1600. In the following, themodeling of the quantization matrix will be described in further detail.

In one embodiment, the present technique uses a single fixed polynomialto represent a quantization matrix, to keep minimal complexity.Modulation of bit cost and complexity is achieved by specifying only then first polynomial coefficients (also called “polynomial parameters”),the remaining ones being implicitly set to zero (or any relevant neutralvalues).

One form of the single fixed polynomial is a fully developed polynomialin (x,y), where x, y indicate the coordinates of a given coefficient ina quantization matrix, with monomials (also called “terms”) ordered byincreasing exponent, as shown in Eq. (3) expressing the way aquantization matrix coefficient M(x,y) is derived as a function ofpolynomial parameters P_(i).

$\begin{matrix}{{M\left( {x,y} \right)} = {\sum\limits_{i}{P_{i} \cdot {m_{i}\left( {x,y} \right)}}}} & (3)\end{matrix}$

where P_(i) are the polynomial coefficients, and m_(i)(x,y)=x^(px) ^(i)y^(py) ^(i) are the terms, px_(i) and py_(i) being the exponents of xand y for monomial m_(i).

This form provides high flexibility with reasonable complexity,controlled by the number of polynomial coefficients that are specified.Since higher exponents are the last ones, reducing the polynomial numberof coefficients reduces de facto the degree of the polynomial, hence itscomplexity. Note that the exponent on a variable (i.e., x or y) in aterm (i.e., m_(i)(x,y)) is called the degree of that variable in thatterm; the degree of the term is the sum of the degrees of the variablesin that term, and the degree of a polynomial is the largest degree ofany one term with non-zero coefficient.

$\begin{matrix}\begin{matrix}{{{More}{specifically}},{{M\left( {x,y} \right)} = {P_{0} +}}} & & & & & & & \\{P_{1}x} & + & {P_{2}y} & + & & & & \\{P_{3}{xy}} & + & {P_{4}x^{2}} & + & {P_{5}y^{2}} & + & & \\{P_{6}x^{2}y} & + & {P_{7}{xy}^{2}} & + & {P_{8}x^{3}} & + & {P_{9}y^{3}} & + \\{P_{10}x^{2}y^{2}} & + & {P_{11}x^{3}y} & + & {P_{12}{xy}^{3}} & + & {{P_{13}x^{4}} + {P_{14}y^{4}}} & + \\\ldots & & & & & & & \end{matrix} & (4)\end{matrix}$

Here monomials are sorted by:

-   -   Rule 1. Increasing degree of the term;    -   Rule 2. Increasing maximal (x or y) exponent; and    -   Rule 3. Increasing y exponent.

In Eq. (4), different subsets of polynomial coefficients define M(x,y)at different degrees. For example, line 1 (“P₀”) defines a degree-0polynomial (i.e., constant), lines 1 and 2 (“P₀+P₁x+P₂y”) define adegree-1 polynomial, lines 1-3 (P₀+P₁x+P₂y+P₃xy+P₄x²+P_(s)y²) define adegree-2 polynomial (like JCTVC-H0460), etc. Each line adds the termsfor the next degree. Note that the maximal number of coefficients fordegrees 0, 1, 2, 3, 4, 5, are respectively 1, 3, 6, 10, 15, and 21.

The series can be continued, but we suggest stopping at degree 4 (i.e.,15 parameters, P₀-P₁₄), because higher degrees need more and moreparameters, with increasing complexity and challenging dynamic range forintermediate computations.

Note that the polynomial may take different forms. For example, rule 2can be ignored, or rule 3 can be modified to follow increasing xexponent. While different forms of M (x,y) can be used, the encoder anddecoder should both have the knowledge of the ordered sequence of themonomials m_(i)(x,y), i=0, 1, 2 . . . Subsequently, when the polynomialparameters are transmitted or received, the i-th polynomial coefficientP_(i) corresponds to the i-th monomial m_(i)(x,y), and thus thepolynomial can be constructed, by associating the polynomial coefficientand the monomial at the same index (i.e., pairing P_(i) and m_(i)(x,y)),as M(x,y)=Σ_(i)P_(i)·m_(i)(x,y).

Here, by using a well-defined transmission (or storage) order ofpolynomial parameters, the complexity of the quantization matrix can becontrolled easily through the number of polynomial parameters used torepresent the quantization matrix. In one example, the number ofpolynomial parameters is transmitted, explicitly or implicitly, in thebitstream as part of the syntax.

Default polynomial coefficients can be used, or the polynomialcoefficients can be transmitted in the bitstream. In one example, alimited number of polynomial parameters can be specified, and theremaining ones can be inferred to be zero. Transmitting fewer parameterstypically needs fewer bits, and a simpler shape that is easy to compute,and more parameters mean more bits, and more complex shape that isharder to compute.

With a degree-4, the number of polynomial parameters can go from 0 to15, which can be coded with 4 bits if it needs to be transmitted. 0 canbe interpreted as default matrix (or to default parameters).

In one embodiment, a symmetry flag, e.g., sym, can be added to specifythe same coefficient for monomials that are symmetrical in x and y, thusforming a symmetric polynomial, with a reduced number of parameters(e.g., 9 instead of 15 for degree 4). The mapping is shown in Table 4for degree-4.

TABLE 4 Mapping of symmetrical polynomial parameters 1 x y xy x² y² x²yxy² x³ y³ x²y² x³y xy³ x⁴ y⁴ sym = 0 P₀ P₁ P₂ P₃ P₄ P₅ P₆ P₇ P₈ P₉ P₁₀P₁₁ P₁₂ P₁₃ P₁₄ sym = 1 P′₀ P′₁ P′₁ P′₂ P′₃ P′₃ P′₄ P′₄ P′₅ P′₅ P′₆ P′₇P′₇ P′₈ P′₈

Normalization of x and y

In one embodiment, we propose to normalize the x and y variables, whichwas not done in previous work like [JCTVC-G352] or [JCTVC-H0460], to:

1—Equalize dynamic range between the various polynomial coefficients. InTable 2, taken from previous work, coefficients a and d have verydifferent range, which can lead to severe problems if increasingpolynomial degree and/or matrix size. With normalization, this problemis solved, and polynomial coefficients can be transmitted with a fixednumber of bits. This is simpler, and has proven more efficient in ourtests than exp-Golomb coding of parameters without x and ynormalization, for the same accuracy on the resulting matrix.

2—Use the same polynomial coefficients for different matrix sizes,including rectangular ones. With this method, using the same polynomialfor a half-size matrix is exactly equivalent to take every othercoefficient of the full-size one; this works in x direction, y, or both.

Let us define a normalization value N, so that

$\begin{matrix}{{x = {N\frac{x^{\prime}}{{size}_{x}}}},{y = {N\frac{y^{\prime}}{{size}_{y}}}}} & (5)\end{matrix}$

with x′,y′ integers in the interval [0 . . . size_(x/y)−1] (i. e., theindices of the columns and rows of the matrix), where size_(x) andsize_(y) are horizontal and vertical size of the matrix.

FIG. 18 illustrates method 1800 for normalizing the matrix coordinates,according to an embodiment. Method 1800 can be implemented inquantization matrix generate 1620, or can be used for performing step1720.

Specifically, for the matrix coordinate x′, based on the horizontal sizeof the quantization matrix and the normalization value N, xnormalization (1810) can be performed:

$x = {N{\frac{x^{\prime}}{{size}_{x}}.}}$

similarly, for the matrix coordinate y′, based on the vertical size ofthe quantization matrix and the normalization value N, y normalization(1830) can be performed:

$y = {N{\frac{y^{\prime}}{{size}_{y}}.}}$

Then the normalized matrix coordinates x and y can be used in modelling(1820) the quantization matrix, for example, used in Eq. (4) to generateM(x,y).

An obvious choice for N is 1, since this makes every monomial in the [0. . . 1) range, so the maximal impact of the variation (accuracy) ofeach coefficient is the same. This is an important consideration whentrying to define the number of significant bits needed. However, thedynamic range of polynomial coefficients to match a given shape varies alot, depending on the number of polynomial coefficients, and on the rank(index i in P,) of each coefficient. (y^(th) and 1^(st) degreepolynomial coefficients tend to be smaller than others, and a highernumber of polynomial coefficients yield bigger dynamic range for all ofthem.

Unconstrained least squares fitting has been performed, for example, byminimizing mean squared error between reconstructed matrix and testmatrix, on a test set based on H.264, HEVC, and other test 8×8 matrices.In FIG. 19 , the left figure shows the maximum absolute value of eachcoefficient, for several experiments with 6, 10, 15, 21 polynomialcoefficients; the right figure shows the maximum absolute value amongall polynomial coefficients, for different numbers of coefficients (3 to21).

Theoretically, polynomial coefficients could be transmitted withunlimited range, using exp-golomb coding. But large polynomialcoefficients would be unrealistic for use in matrix computation, whichshall be fully specified, thus bit-limited. This implies defining therange of polynomial coefficients and their accuracy.

Since quantization matrices are typically defined as 8-bit numbers, itmakes sense to define polynomial coefficients with a similar bit depth.If the number of polynomial coefficients is limited to 15 (i.e., degree4), they could be constrained to [−512 . . . 511] range (i.e., 10-bitsigned), and dropping 2 LSBs (Least Significant Bits) would make them8-bit signed. This would reduce accuracy of matrix (0,0) value by afactor 4, but overall accuracy can be satisfying because eachcoefficient affects matrix at different places with various levels.

Normalization value N=1 is not the only possible choice, and othervalues yield other compromises on coefficient range and accuracy: forN>1, accuracy of higher-index coefficients have greater impact, as shownon FIG. 20 . Impact of coefficient bounds (i.e., dynamic range) onfitting quality for various N is shown on FIG. 21 , suggesting 512, 256,and 128 for N=1, √{square root over (2)}, 2.

A recommended N is 2, because it yields stable low coefficient rangewith a good compromise on overall accuracy, and keeps full accuracy for(0,0) matrix position, which is valuable.

Efficient Computation

Since size_(x) and size_(y) are typically powers of 2, and if N is alsoa power of two (we recommend N=2), the division in Eq. (5) will simplifyto bit shifts. Let sx=log2 (size_(x))−log2 (N) and sy=log2(size_(y))−log2 (N), then equation (5) can be written as:

$\begin{matrix}{{x = \frac{x^{\prime}}{2^{sx}}},{y = \frac{y^{\prime}}{2^{sy}}}} & (6)\end{matrix}$

Reported in Eq. (3), this becomes:

$\begin{matrix}{{M\left( {x^{\prime},y^{\prime}} \right)} = {{\sum\limits_{i}{{P_{i}\left( \frac{x^{\prime}}{2^{sx}} \right)}^{{px}_{i}}\left( \frac{y^{\prime}}{2^{sy}} \right)^{{py}_{i}}}} = {\sum\limits_{i}{P_{i}\frac{x^{\prime{px}_{i}}y^{\prime{py}_{i}}}{2^{{{sx}.{px}_{i}} + {{sy}.{py}_{i}}}}}}}} & (7)\end{matrix}$

Now if defining smax=max(sx. px+sy. py), we can write

$\begin{matrix}{{M\left( {x^{\prime},y^{\prime}} \right)} = \frac{\sum_{i}{P_{i}x^{\prime{px}_{i}}y^{\prime{py}_{i}}\text{.2}^{{smax} - {{sx}.{px}_{i}} - {{sy}.{py}_{i}}}}}{2^{smax}}} & (8)\end{matrix}$

With m′_(i)(x′,y′)=x′^(px) ^(i) y′^(py) ^(i) ,

$\begin{matrix}{{M\left( {x^{\prime},y^{\prime}} \right)} = \frac{\sum_{i}{P_{i}{m_{i}^{\prime}\left( {x^{\prime},y^{\prime}} \right)}\text{.2}^{{smax} - {{sx}.{px}_{i}} - {{sy}.{py}_{i}}}}}{2^{smax}}} & (9)\end{matrix}$

Since smax≥0 and smax−sx. px_(i)−sy. py_(i)≥0, the multiplication by2^(smax−sx. px) ^(i) ^(−sy. py) ^(i) simplifies to left shifting, andthe final division by 2^(smax) can be implemented with a right shift,preferably with rounding:

$\begin{matrix}{{{{M\left( {x^{\prime},y^{\prime}} \right)} = \left( {\left( {\sum\limits_{i}{P_{i}{m_{i}^{\prime}\left( {x^{\prime},y^{\prime}} \right)}{s_{i}}}} \right) + {rnd}} \right)}}{smax}} & (10)\end{matrix}$

with s_(i)=smax−(sx. px_(i)+sy. py_(i)) and rnd=1<<(smax−1).

This method retains good accuracy, with integer computations. FIG. 22illustrates quantization matrix generator 2200 with integer computing,according to an embodiment. Generator 2200 can be used as module 1620.

From the matrix size (size_(x), size_(y)) and the normalization value N,sx=log2 (size_(x))−log2(N) and sy=log2(size_(y))−log2(N), and thenormalization data can be derived (2220): s_(i)=smax−(sx. px_(i)+sy.py_(i)). The rounding and shifting value can be derived (2230) as:smax=max(sx. px+sy. py), and rnd=1<<(smax−1).

From quantization matrix coordinate x′ and y′, the i-th monomial can bederived (2210) as: m′_(i)(x′,y′)=x′^(px) ^(i) y′^(py) ^(i) ^(py) ^(i) .Pairing (2240) the i-th monomial with the i-th polynomial parameterP_(i), P_(i)m′_(i)(x′,y′) is formed (2240). Then left shifting isapplied (2250): P_(i)m′_(i)(x′,y′)<<s_(i), and the shifted results aresummed (2260) up: Σ_(i)P_(i)m′_(i)(x′,y′)<<s_(i), rounded by adding(rnd) (2270) and right shifted (2280) by smax to form the elements inthe quantization matrix:M(x′,y′)=((Σ_(i)P_(i)m′_(i)(x′,y′)<<s_(i))+rnd)>>smax.

FIG. 23 illustrates method 2300 for generating the quantization matrixwith integer computing, according to an embodiment. Generator 2300 canbe used as in module 1620 or be performed in step 1720.

Method 2300 starts at step 2305. From the matrix size (size_(x),size_(y)) and the normalization value N, sx=log2(size_(x))−log2(N) andsy=log2(size_(y))−log2(N), the rounding and shifting value can bederived (2310) as: smax=max(sx. px+sy. py), and rnd=1<<(smax−1).

The decoder then initializes (2320) variable i to 0, and M (x′,y′)=rnd.At step 2330, the normalization data can be derived: s_(i)=smax−(sx.px_(i)+sy. py_(i)). From quantization matrix coordinate x′ and y′, thei-th monomial can be derived (2340) as: m′_(i)(x′,y′)=x′^(px) ^(i)y′^(py) ^(i) . Pairing (2350) the i-th monomial with the i-th polynomialparameter P_(i), P_(i)m′_(i)(x′,y′) is formed, and added to M(x,y) afterleft shifting (2350): M(x′,y′)+=P_(i)m′_(i)(x′,y′)<<s_(i). At step 2360,variable i is incremented by 1. At step 2370, the decoder checks whetherthe last monomial has been processed. If not, the control returns tostep 2330. Otherwise, at step 2380, M(x′,y′) is right shifted by smax toform the elements in the quantization matrix: M(x′,y′)>>=smax. Method2300 ends at step 2399.

Most variables can be pre-computed, since rnd and smax only depend onmatrix size, s_(i) depend on matrix size and monomial index. Table 5shows an example:

TABLE 5 Example computation of s_(i), rnd, smax m_(i) 1 x y xy x² y² x²yxy² x³ y³ x²y² x³y xy³ x⁴ y⁴ px_(i) 0 1 0 1 2 0 2 1 3 0 2 3 1 4 0 py_(i)0 0 1 1 0 2 1 2 0 3 2 1 3 0 4

For an 8 × 8 matrix and N = 2: sx = 2, sy = 2, smax = 8, and rnd = 128sx · px_(i) + sy · py_(i) 0 2 2 4 4 4 6 6 6 6 8 8 8 8 8 s_(i) 8 6 6 4 44 2 2 2 2 0 0 0 0 0

It can be noted that m′_(i) do not depend on matrix size, hence they canbe computed once lit for all, with a subpart used for smaller sizes.Efficient incremental implementations are possible, with very fewmultiplications.

Alternatively, shifting by s_(i) can be applied to P_(i) before thecomputation of the matrix, so that the latter is a series ofmultiply-accumulate, initialized by rnd, and followed by a right shift,which is a very commonplace operation.

Analysis of Intermediate Computation Bit Depth

For the example above (8×8 matrix with N=2), with 8-bit signed P_(i),Table 6 shows the bit depth of m′_(i), P_(i)m′_(i) andP_(i)m′_(i)<<s_(i).

TABLE 6 i 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 m′_(i) bitdepth 0 3 3 6 66 9 9 9 9 12 12 12 12 12 P_(i)m′_(i) → 8 11 11 14 14 14 17 17 17 17 2020 20 20 20 P_(i)m′_(i) << S_(i) → 16 17 17 18 18 18 19 19 19 19 20 2020 20 20

However, we expect a result with mbd (matrix bit depth)=8 bits(unsigned) for the matrix values, after shifting right by smax=8 bits.P_(i) shall be constrained at design time so that the result is positiveand within bounds (0-255). Furthermore, truncation (of upper bits) aftershifting (mbd=8 bits), or before (mbd+smax=16 bits), is equivalent.Also, for an addition, truncating operands make no difference if theresult is truncated to the same bit depth, and two's complementrepresentation of signed values makes this applicable to both signed andunsigned numbers.

This means that the accumulator of P_(i)m′_(i)<<s_(i) can be limited to(mbd+smax)=16 bits. However, multipliers cannot be reduced (but theirresult can be truncated to the working bit depth). In the 8×8 example,they must have 12 and 8-bit inputs and 16-bit result; for a 256×256matrix, they should have 32 and 8-bit inputs, and 36-bit result.

The final right shift could be adjusted, to give some flexibility on therange and accuracy of P_(i). Eq. (10) is then modified as follows:

$\begin{matrix}{{{{M\left( {x^{\prime},y^{\prime}} \right)} = \left( {\left( {\sum\limits_{i}{P_{i}{m_{i}^{\prime}\left( {x^{\prime},y^{\prime}} \right)}{s_{i}}}} \right) + {rnd}} \right)}}\left( {{smax} - s_{base}} \right)} & (11)\end{matrix}$

This is intended to be used with positive values, to improve accuracy ofthe parameters. to If negative values are used to increase dynamicrange, it should be noted that this will have an impact on intermediatecomputation bit depth (for example, −1 will add 1 bit the accumulatorbit depth).

Rounding value rnd should be adjusted accordingly:rnd=1<<(smax−s_(base)−1).

Lower clipping can be added on top of polynomial representation, asshown in Eq. (12) (this is also applicable on top of Eq. (10) or (11)).This can improve fitting to usual quantization matrices because theyoften have a flat area on the lower end.

$\begin{matrix}{{M\left( {x,y} \right)} = {\max\left( {{\sum\limits_{i}{P_{i}{m_{i}\left( {x,y} \right)}}},{clip}} \right)}} & (12)\end{matrix}$

This has a significant impact on fitting quality, especially for smallnumber of parameters, as illustrated in FIG. 24 and FIG. 25 (testedwithout symmetry flag nor custom shifting). In particular, the leftfigure in FIG. 24 shows the fitting for the default HEVC intra matrixusing 10-parameters, without clipping, and the right figure is withclipping. The figures in FIG. 25 shows the impact of clipping on fittingerror for the default HEVC intra 8×8 matrix (left), JCTVC-F475 8×8asymmetric test matrix (middle), H.264 matrix (right).

It should be noted that discussions around polynomial result range areno more applicable on the lower side, because of clipping: polynomial isnow unbounded below zero, since clipping will correct it. Either thepolynomial coefficients should be constrained to give result withinmatrix range, or the result bit depth should be increased to accommodatefor the bigger range, with relaxed constraint. The recommended option isto have 1 bit more, to allow negative polynomial (up to −256 for 8-bitmatrix coefficients); clipping would make the result positive and dropthe sign bit.

Current HEVC syntax is reproduced in the following (see HEVC standardversion 3, April 2015):

scaling_list_data( ) { Descriptor   . . .   nextCoef = 8   coefNum =Min( 64, ( 1 << ( 4 + ( sizeId << 1 ) ) ) )   if( sizeld > l ) {   scaling_list_dc_ minus8[ sizeId - 2 ][ matrixId ] se(v)    nextCoef =scaling_list_dc_coef_minus8[ sizeId − 2 ][ matrixId ] + 8   }   for( i =0; i < coefNum; i++) {    scaling_list_delta_coef se(v)    nextCoef = (nextCoef + scaling_list_delta coef + 256 ) % 256    ScalingList[ sizeId][ matrixId ][ i ] = nextCoef   }   . . . }

Based on the proposed model, this portion of syntax can be replaced forexample by:

scaling_list_data( ) { Descriptor   . . .   scaling_poly_nb_param u(4)  if( scaling_poly_nb_param > 0 ) {    scaling_poly_symmetric f(1)   scaling_poly_shift[ sizeId ][ matrixId ] u(2)    scaling_poly_clip[sizeId ][ matrixId ] f(4)    for( i = 0; i < scaling_poly_nb_param; i++){     scaling_poly_param[ i ] i(8)    }   }   . . . }

scaling_poly_nb_param=0 means “default parameters”. In that case thefollowing syntax elements are skipped. Otherwise, it defines the numberof scaling_poly_param present to in the syntax. Please note that it maybe taken from a list of predefined values. The number of parameters mayalso be indicated using a look-up table, for example, a tablenb_param[i][sym] may be known at both the encoder and decoder, and indexi of the table is signaled in the bitstream. In one example, the numberof parameters is restricted to 1, 3, 4, 6, 8, 10, 11, 13, and 15.

scaling_poly_symmetric, if 1, indicates that scaling_poly_param shouldbe repeated for x/y symmetric polynomial coefficients. This flag can beremoved if this feature is not desired. If removed, it should beinferred to 0 in the following.

scaling_poly_shift selects s_(base) from a set of predefined values. Forexample, s_(base)=scaling_poly_shift—offset, with offset selected inaccordance to normalization. For N=2, we recommend 0 to 3 range, i.e.,use s_(base)=scaling_poly_shift.

scaling_poly_clip defines the lower clipping value from a range ofpredefined values. It can take fewer bits than shown in the table, forexample, it can use 2 or 3 instead of 4 bits. This could be [1, 4, 8,16], for example. Please note that the lower clipping value typically isno greater than 16 (the neutral value for 8-bit matrix coefficients usedas quantization scaling factors as in HEVC).

scaling_poly_param is used to represent the polynomial coefficients forthe selected matrix (scaling_poly_coef, that matches P_(i) in equations(3), (11), (12), and others).

The size, order, and format of syntax elements can be changed, and someof them can depend on scaling_poly_nb_param. For example,scaling_poly_symmetric is useless if scaling_poly_nb_param is 0 or 1,and the number of bits of scaling_poly_shift can increase with thenumber of parameters, etc. In addition, the syntax can vary, andpolynomial coefficients could be derived in an indirect way.

In HEVC, the scaling list is signaled in SPS (Sequence Parameter Set) orPPS (Picture

Parameter Set). It should be noted that the present embodiments are notlimited to transmitting the quantization matrix information in SPS orPPS. For example, the quantization matrix information can be transmittedin a parameter set dedicated to transmitting matrices. In anotherexample, the quantization matrix information can be transmitted at theblock level, possibly with the number of parameters being variable.

The following algorithm can be used to duplicate symmetric elements onthe fly when needed:

  k = n = degree = 0; for( i = 0; i < scaling_poly_nb_param; i++ ) { scaling_poly_coef[sizeId] [matrixId]  [n++] = scaling_poly_param[i] ; if( scaling_poly_symmetric ) {   k++;   if( k > (degree+1) % 2 ) {   scaling_poly_coef[sizeId] [matrixId]    [n++] = scaling_poly_param[i];    k++;   }   if( k > degree ) {    degree ++;    k = 0;   }  } }

With this method, there is no need to save symmetric flag because allcoefficients are derived on the fly. Note that this is a simple copywhen scaling_poly_symmetric is zero.

Note that non-specified parameters should have no effect: eitherscaling_poly_coef should be initialized to zero, or n should be saved sothat matrix computation can terminate early.

This algorithm can be written in tabular form as follows:

scaling_list_data( ) { Descriptor   . . .   scaling_poly_nb_param u(4)  if( scaling_poly_nb_param > 0 ) {    scaling_poly_symmetric f(1)   scaling_poly_shift[ sizeId ][ matrixId ] u(2)    scaling_poly_clip[sizeId ][ matrixId ] f(4)    k = n = degree = 0;    for( i = 0; i <scaling_poly_nb_param; i++) {     scaling_poly_param i(8)    scaling_poly_coef[ sizeId ][ matrixId ][ n++ ] = scaling_poly_param    k++     if( scaling_poly_symmetric && k > (degree+1) % 2 ) {     scaling_poly_coef[ sizeId ][ matrixId ][ n++ ] =         scaling_poly_param      k++     }     if( k > degree ) {     degree++      k = 0     }    }   }   . . . }

Residuals

As in [JCTVC-H0460], residuals can be added to further improve the fitto a given matrix shape. To optimize coding cost, the number ofresiduals should be variable, with a scanning order followinglow-frequency to high-frequency coefficients order, such as up-rightdiagonal (as in HEVC) or zig-zag (as in AVC), because low-frequencycoefficients are often regarded as more critical. Non-coded residualsare inferred to 0.

This can be coded with the following syntax, immediately afterpolynomial definition:

scaling_list_data( ) { Descriptor   . . .   scaling_residual_nb ue(v)  for( i = 0; i < scaling_residual_nb; i++) {    scaling_residual_list[sizeId ][ matrixId ][i] se(v)   }   . . . }

In another embodiment, residuals may not be recommended, because ofcomplexity and memory considerations.

Matrix as QP Offset

As in [JCTVC-I0284], quantization matrices can be used as QP offsets(denoted as “QP-offset mode” or “log scale mode” because QP representsquantizer scale in a log scale) rather than scaling factors (denoted as“linear scale mode”), since it brings several advantages:

-   -   Simpler dequantization, by removal of a multiplication, which in        turn reduces computation dynamic range. This leaves room for        increased accuracy for levScale, which allows its use as        √{square root over (2)} factor (using an additional QP offset)        for some non-square transform normalization, and completely        removes multiplications other than levScale.    -   Quantization is similarly simplified by the removal of the        division by quantization matrix coefficient (usually implemented        as a multiplication by inverse of quantization matrix, with        accuracy compromise, memory impact, and even more pressure on        dynamic range than for dequantization), leaving only        multiplication by inverse of levScale.    -   It makes sense to unify the representation of all quantization        adjustments    -   Neutral value (flat matrix) is zero    -   Log scale naturally provides better accuracy to low frequency        coefficients, which are more critical    -   HEVC default matrix formula uses an exponent; log scale        representation should be easier to fit with a polynomial. In        general, fit is expected to be better in log scale, further        reducing the need for residual or a high number of parameters,        or even clipping.    -   Exponentiation would turn to a multiplication, which is much        easier to implement    -   Log scale representation being signed (e.g., −128 to +127),        dynamic range extension related to clipping may not be needed    -   Increased QP granularity, required for smooth matrix definition,        also provides better rate control

The drawback is the need for finer QP granularity, which impacts atleast

-   -   levScale definition,    -   delta-QP bit cost for slice header: would barely add two bits    -   delta-QP bit cost for coding units. This can be mitigated by        introducing a delta-QP scale (in PPS, or slice header), as        already proposed in the past (e.g., [JCTVC-C135], D. Hoang,        “Flexible scaling of quantization parameter,” JCTVC-C135, JCT-VC        3rd Meeting: Guangzhou, CN, Oct. 7-15, 2010), which can actually        decrease delta-QP coding cost.    -   threshold tables used by deblocking filter    -   Lagrange multipliers used at various places

Please note that the impact can be limited to levScale if QP granularityis increased for quantization matrices only, as in [JCTVC-I0284].

We further recommend adopting a QP scale with 1/16 step (as in[JCTVC-A114], J. Jung et al, “Description of video coding technologyproposal by France Telecom, NTT, NTT DOCOMO, Panasonic and Technicolor”,JCTVC-A114, JCT-VC 1st Meeting: Dresden, Del., Apr. 15-23, 2010) insteadof ⅙, or 1/12 as in [JCTVC-I0284]. This further simplifies thequantization/dequantization operations, by removing division by 6 andmodulo-6 operations, which are replaced by bit mask and shift (no actualoperation in hardware). This also eases the implementation of QPoffsets.

Then, QP would generally use two more bits as a result of increasedaccuracy. An appropriate conversion formula can be defined to convert QPvalues (let us call them QP₆) to the new standard (QP16). Quantizationmatrix bit depth can be decreased to 7 bits signed: −64 to +63 rangematches the 1/16 to 16 range of linear-scale matrix definition, with 30%better resolution around the neutral value. This in turn reduces the bitdepth of polynomial parameters to 7-bit, leading to lower coding cost,and reduces matrix computation dynamic range as discussed above.

The syntax would be changed to the following (scaling_poly_param reducedto 7-bit):

scaling_list_data( ) { Descriptor   . . .   scaling_poly_nb_param u(4)  if( scaling_poly_nb_param > 0 ) {    scaling_poly_symmetric f(1)   scaling_poly_shift[ sizeId ][ matrixId ] u(2)    scaling_poly_clip[sizeId ][ matrixId ] f(4?)    for( i = 0; i < scaling_poly_nb_param;i++) {     scaling_poly_param[ i ] i(7)    }   }   . . . s}

FIG. 26 illustrates the fitting results for the default HEVC intramatrix with 6 parameters. The test is performed without symmetry, shift,nor clipping. The left figure shows the result from using thequantization matrix for linear scaling (51 bits used to encode thequantization matrix), and the right figure shows the result from usingthe quantization matrix for QP offsets (46 bits used to encode thequantization matrix). Overall, in various tests, fit generally looksbetter in QP-offset mode, even when evaluated in linear domain.

Default Parameters

Example parameters to match default HEVC 8×8 intra matrix can be foundbelow:

Linear Scale Mode:

-   -   clip=16, Sbase=0, sym=1    -   5-parameters (8-bit) P_(i)=[−53, 94, −121, −29, 36]    -   7-parameters (8-bit) P_(i)=[6, 8, 18, 0,−17, 1, 21]

QP-Offset Mode:

-   -   clip=0, Sbase=0, sym=1    -   4-parameters (7-bit) P_(i)=[−2, −2, 10, 4]    -   5-parameters (7-bit) P_(i)=[−27, 32, −26, −7, 8]    -   6-parameters (7-bit) P_(i)=[−8, 2, −3, 11, 3, −4]    -   7-parameters (7-bit) P_(i)=[1, −17, 38, 21, −16, −5, 9]    -   8-parameters (7-bit) P_(i)=[−3, −1, 6, 3, 4, 1, 6, −5]    -   9-parameters (7-bit) P_(i)=[0, −8, 10, 14, 3, −7, 6, −5, 2]

Coding default HEVC intra 8×8 matrix with HEVC syntax would take 265bits (1 bit to code scaling_list_pred_mode_flag, and 264 bits to codethe scaling list delta coef values).

For the linear scale mode and QP-offset mode, the coding bit cost andaverage fitting absolute error for the suggested parameters are analyzedas follows:

-   -   4 bits for scaling_poly_nb_param,    -   1 bit for scaling_poly_symmetric,    -   2 bits for scaling_poly_shift,    -   4 bits for scaling_poly_clip, and    -   8 or 7 bits for each parameter.

To enable error comparison, error for QP offset mode is computed inlinear demain, i.e., by first converting back the QP offsets to scalefactors. LF error is the average absolute error for low frequencies half(x+y<8). It can be seen from Table 7 that using 51 bits for the linearscale mode, and 46 bits for the QP offset mode, the error forrepresenting the quantization matrix is small. Thus, compared to 265bits used by the HEVC standard, the proposed methods achieved a good bitsaving in encoding the quantization matrix without a big loss inaccuracy. In addition, as also shown in Table 7 the method can be easilyscaled, by using more polynomial parameters, to improve the accuracy inrepresenting the quantization matrix.

TABLE 7 Number of Linear scale QP offset parameters Bits Error LF errorBits Error LF error 4 39 1.23 0.60 5 51 0.89 0.89 46 1.26 0.58 6 53 0.790.48 7 67 0.50 0.36 60 0.57 0.43 8 67 0.59 0.34 9 74 0.66 0.23

Polynomial coefficients can be obtained by least squares fitting to anexisting 8×8 matrix, with constraints on coefficient range, andresulting matrix values. If relevant, lower clipping can be setmanually.

In some embodiments, the polynomial can take other forms than what isdescribed above, or be a combination of several sub-polynomials:

-   -   x and y can be replaced by u and v, with u=(x+y) and v=(x−y), or        even u=(ax+by) and v=(bx−ay), with a+b=1, and a possibly        transmitted. For flat-diagonal matrices, all coefficients in v        would be zero, and for symmetric matrices (with respect to u        axis), all coefficients with odd powers of v would be zero.        However, the benefits may be difficult to take advantage of, and        an additional parameter (a) may be required.    -   Two separable polynomials may be combined to generate a matrix,        for example, M(x,y)=P₁(x)·P₂(y) or M(x,y)=P₁(x)+P₂(y), with x        and y possibly replaced by u and v defined above, and with P₁        and P₂ defined separately with possibly a different number of        parameters for each.

Various methods as described above may be used to modify, e.g., thequantization module and de-quantization module (130, 140, 240) of theencoder 100 and decoder 200 as shown in FIG. 1 and FIG. 2 ,respectively. Moreover, the present embodiments are not limited to VVCor HEVC, and may be applied to other standards, recommendations, andextensions thereof.

Various methods are described herein, and each of the methods comprisesone or more steps or actions for achieving the described method. Unlessa specific order of steps or actions is required for proper operation ofthe method, the order and/or use of specific steps and/or actions may bemodified or combined. Unless indicated otherwise, or technicallyprecluded, the aspects described in this application can be usedindividually or in combination. Various numeric values are used in thepresent application, for example, the parameter used in bit shifting forinteger implementation, and the example polynomial parameters. Thespecific values are for example purposes and the aspects described arenot limited to these specific values.

FIG. 27 illustrates a method (2700) of encoding video data, according toan embodiment. At step 2710, a parametric model based on a sequence ofparameters is accessed by the encoder. At step 2720, the encoderdetermines a plurality of parameters to model the quantization matrix.At step 2730, the encoder associates each parameter of the plurality ofparameters with a corresponding parameter of a subset of the sequence ofparameters, to represent the quantization matrix. At step 2740, theencoder quantizes transform coefficients of a block of an image based onthe quantization matrix. At step 2750, the encoder encodes quantizedtransform coefficients, for example, using an entropy encoder.

FIG. 28 illustrates a method (2800) of decoding video data, according toan embodiment. At step 2810, a parametric model based on a sequence ofparameters is accessed by the decoder. At step 2820, the decoderdetermines a plurality of parameters to model the quantization matrix.At step 2830, the decoder associates each parameter of the plurality ofparameters with a corresponding parameter of a subset of the sequence ofparameters, to represent a quantization matrix. At step 2840, thedecoder de-quantizes transform coefficients of a block of an image basedon the quantization matrix. At step 2850, the decoder reconstructs theblock of the image responsive to the de-quantized transformcoefficients.

FIG. 29 illustrates a block diagram of an example of a system in whichvarious aspects and embodiments are implemented. System 2900 can beembodied as a device including the various components described belowand is configured to perform one or more of the aspects described inthis application. Examples of such devices, include, but are not limitedto, various electronic devices such as personal computers, laptopcomputers, smartphones, tablet computers, digital multimedia set topboxes, digital television receivers, personal video recording systems,connected home appliances, and servers. Elements of system 2900, singlyor in combination, can be embodied in a single integrated circuit,multiple ICs, and/or discrete components. For example, in at least oneembodiment, the processing and encoder/decoder elements of system 2900are distributed across multiple ICs and/or discrete components. Invarious embodiments, the system 2900 is communicatively coupled to othersystems, or to other electronic devices, via, for example, acommunications bus or through dedicated input and/or output ports. Invarious embodiments, the system 2900 is configured to implement one ormore of the aspects described in this application.

The system 2900 includes at least one processor 2910 configured toexecute instructions loaded therein for implementing, for example, thevarious aspects described in this application. Processor 2910 caninclude embedded memory, input output interface, and various othercircuitries as known in the art. The system 2900 includes at least onememory 2920 (e.g., a volatile memory device, and/or a non-volatilememory device). System 2900 includes a storage device 2940, which caninclude non-volatile memory and/or volatile memory, including, but notlimited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic diskdrive, and/or optical disk drive. The storage device 2940 can include aninternal storage device, an attached storage device, and/or a networkaccessible storage device, as non-limiting examples.

System 2900 includes an encoder/decoder module 2930 configured, forexample, to process data to provide an encoded video or decoded video,and the encoder/decoder module 2930 can include its own processor andmemory. The encoder/decoder module 2930 represents module(s) that can beincluded in a device to perform the encoding and/or decoding functions.As is known, a device can include one or both of the encoding anddecoding modules. Additionally, encoder/decoder module 2930 can beimplemented as a separate element of system 2900 or can be incorporatedwithin processor 2910 as a combination of hardware and software as knownto those skilled in the art.

Program code to be loaded onto processor 2910 or encoder/decoder 2930 toperform the various aspects described in this application can be storedin storage device 2940 and subsequently loaded onto memory 2920 forexecution by processor 2910. In accordance with various embodiments, oneor more of processor 2910, memory 2920, storage device 2940, andencoder/decoder module 2930 can store one or more of various itemsduring the performance of the processes described in this application.Such stored items can include, but are not limited to, the input video,the decoded video or portions of the decoded video, the bitstream,matrices, variables, and intermediate or final results from theprocessing of equations, formulas, operations, and operational logic.

In several embodiments, memory inside of the processor 2910 and/or theencoder/decoder module 2930 is used to store instructions and to provideworking memory for processing that is needed during encoding ordecoding. In other embodiments, however, a memory external to theprocessing device (for example, the processing device can be either theprocessor 2910 or the encoder/decoder module 2930) is used for one ormore of these functions. The external memory can be the memory 2920and/or the storage device 2940, for example, a dynamic volatile memoryand/or a non-volatile flash memory. In several embodiments, an externalnon-volatile flash memory is used to store the operating system of atelevision. In at least one embodiment, a fast external dynamic volatilememory such as a RAM is used as working memory for video coding anddecoding operations, such as for MPEG-2, HEVC, or VVC (Versatile VideoCoding).

The input to the elements of system 2900 can be provided through variousinput devices as indicated in block 2905. Such input devices include,but are not limited to, (i) an RF portion that receives an RF signaltransmitted, for example, over the air by a broadcaster, (ii) aComposite input terminal, (iii) a USB input terminal, and/or (iv) anHDMI input terminal.

In various embodiments, the input devices of block 2905 have associatedrespective input processing elements as known in the art. For example,the RF portion can be associated with elements suitable for (i)selecting a desired frequency (also referred to as selecting a signal,or band-limiting a signal to a band of frequencies), (ii) downconvertingthe selected signal, (iii) band-limiting again to a narrower band offrequencies to select (for example) a signal frequency band which can bereferred to as a channel in certain embodiments, (iv) demodulating thedownconverted and band-limited signal, (v) performing error correction,and (vi) demultiplexing to select the desired stream of data packets.The RF portion of various embodiments includes one or more elements toperform these functions, for example, frequency selectors, signalselectors, band-limiters, channel selectors, filters, downconverters,demodulators, error correctors, and demultiplexers. The RF portion caninclude a tuner that performs various of these functions, including, forexample, downconverting the received signal to a lower frequency (forexample, an intermediate frequency or a near-baseband frequency) or tobaseband. In one set-top box embodiment, the RF portion and itsassociated input processing element receives an RF signal transmittedover a wired (for example, cable) medium, and performs frequencyselection by filtering, downconverting, and filtering again to a desiredfrequency band. Various embodiments rearrange the order of theabove-described (and other) elements, remove some of these elements,and/or add other elements performing similar or different functions.Adding elements can include inserting elements in between existingelements, for example, inserting amplifiers and an analog-to-digitalconverter. In various embodiments, the RF portion includes an antenna.

Additionally, the USB and/or HDMI terminals can include respectiveinterface processors for connecting system 2900 to other electronicdevices across USB and/or HDMI connections. It is to be understood thatvarious aspects of input processing, for example, Reed-Solomon errorcorrection, can be implemented, for example, within a separate inputprocessing

IC or within processor 2910 as necessary. Similarly, aspects of USB orHDMI interface processing can be implemented within separate interfaceICs or within processor 2910 as necessary. The demodulated, errorcorrected, and demultiplexed stream is provided to various processingelements, including, for example, processor 2910, and encoder/decoder2930 operating in combination with the memory and storage elements toprocess the datastream as necessary for presentation on an outputdevice.

Various elements of system 2900 can be provided within an integratedhousing, Within the integrated housing, the various elements can beinterconnected and transmit data therebetween using suitable connectionarrangement 2915, for example, an internal bus as known in the art,including the I2C bus, wiring, and printed circuit boards.

The system 2900 includes communication interface 2950 that enablescommunication with other devices via communication channel 2990. Thecommunication interface 2950 can include, but is not limited to, atransceiver configured to transmit and to receive data overcommunication channel 2990. The communication interface 2950 caninclude, but is not limited to, a modem or network card and thecommunication channel 2990 can be implemented, for example, within awired and/or a wireless medium.

Data is streamed to the system 2900, in various embodiments, using aWi-Fi network such as IEEE 802.11. The Wi-Fi signal of these embodimentsis received over the communications channel 2990 and the communicationsinterface 2950 which are adapted for Wi-Fi communications. Thecommunications channel 2990 of these embodiments is typically connectedto an access point or router that provides access to outside networksincluding the Internet for allowing streaming applications and otherover-the-top communications. Other embodiments provide streamed data tothe system 2900 using a set-top box that delivers the data over the HDMIconnection of the input block 2905. Still other embodiments providestreamed data to the system 2900 using the RF connection of the inputblock 2905.

The system 2900 can provide an output signal to various output devices,including a display 2965, speakers 2975, and other peripheral devices2985. The other peripheral devices 2985 include, in various examples ofembodiments, one or more of a stand-alone DVR, a disk player, a stereosystem, a lighting system, and other devices that provide a functionbased on the output of the system 2900. In various embodiments, controlsignals are communicated between the system 2900 and the display 2965,speakers 2975, or other peripheral devices 2985 using signaling such asAV.Link, CEC, or other communications protocols that enabledevice-to-device control with or without user intervention. The outputdevices can be communicatively coupled to system 2900 via dedicatedconnections through respective interfaces 2960, 2970, and 2980.Alternatively, the output devices can be connected to system 2900 usingthe communications channel 2990 via the communications interface 2950.The display 2965 and speakers 2975 can be integrated in a single unitwith the other components of system 2900 in an electronic device, forexample, a television. In various embodiments, the display interface2960 includes a display driver, for example, a timing controller (T Con)chip.

The display 2965 and speaker 2975 can alternatively be separate from oneor more of the other components, for example, if the RF portion of input2905 is part of a separate set-top box. In various embodiments in whichthe display 2965 and speakers 2975 are external components, the outputsignal can be provided via dedicated output connections, including, forexample, HDMI ports, USB ports, or COMP outputs.

According to an embodiment, a method for video decoding is presented,comprising:

accessing a parametric model that is based on a sequence of parameters;determining a plurality of parameters that correspond to a subset ofsaid sequence of parameters; associating each parameter of saidplurality of parameters with a corresponding parameter of said subset ofsaid sequence of parameters, to represent a quantization matrix;de-quantizing transform coefficients of a block of an image based onsaid quantization matrix; and reconstructing said block of said imageresponsive to said de-quantized transform coefficients.

According to another embodiment, a method for video encoding ispresented, comprising: accessing a parametric model that is based on asequence of parameters; determining a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associating eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; quantizing transform coefficients of a block of animage based on said quantization matrix; and entropy encoding quantizedtransform coefficients.

According to another embodiment, an apparatus for video decoding ispresented, comprising one or more processors, wherein said one or moreprocessors are configured to: access a parametric model that is based ona sequence of parameters; determine a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associate eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; de-quantize transform coefficients of a block of animage based on said quantization matrix; and reconstruct said block ofsaid image responsive to said de-quantized transform coefficients. Theapparatus can further comprise one or more memories coupled to said oneor more processors.

According to another embodiment, an apparatus for video encoding ispresented, comprising one or more processors, wherein said one or moreprocessors are configured to: access a parametric model that is based ona sequence of parameters; determine a plurality of parameters thatcorrespond to a subset of said sequence of parameters; associate eachparameter of said plurality of parameters with a corresponding parameterof said subset of said sequence of parameters, to represent aquantization matrix; quantize transform coefficients of a block of animage based on said quantization matrix; and entropy encode quantizedtransform coefficients. The apparatus can further comprise one or morememories coupled to said one or more processors.

According to another embodiment, a signal is formatted to include: avalue indicating a number of parameters in a plurality of parameters;said plurality of parameters, wherein each parameter of said pluralityof parameters is associated with a corresponding parameter of a subsetof a sequence of parameters, to represent a quantization matrix, whereina parametric model is based on said sequence of parameters; andtransform coefficients of a block of an image quantized based on saidquantization matrix.

According to one embodiment, a value is encoded or decoded to indicatethe number of parameters in said plurality of parameters. Usually, asthe number of parameters increases, the quantization matrix becomes morecomplex.

According to one embodiment, the parametric model corresponds to apolynomial in (x, y), where x and y indicate matrix coordinates, saidsequence of parameters corresponding to an ordered sequence ofpolynomial coefficients, said polynomial being defined by said orderedsequence of polynomial coefficients and an ordered sequence of terms.

According to one embodiment, said ordered sequence of polynomialcoefficients are symmetrical in x and y.

According to one embodiment, a set of polynomial coefficients is decodedfrom a bitstream, wherein said ordered sequence of polynomialcoefficients is determined responsive to said decoded set of polynomialcoefficients.

According to one embodiment, a value indicating a degree of saidpolynomial is encoded or decoded, wherein said value indicating thenumber of parameters is obtained from said value indicating said degree.In one example, the degree of said polynomial is set to 4.

According to one embodiment, there is a one-to-one correspondencebetween said ordered sequence of polynomial coefficients and saidordered sequence of terms.

According to one embodiment, said polynomial is represented as

${M\left( {x,y} \right)} = {P_{0} + \begin{matrix}{P_{1}x} & + & {P_{2}y} & + & & & & & & \\{P_{3}{xy}} & + & {P_{4}x^{2}} & + & {P_{5}y^{2}} & + & & & & \\{P_{6}x^{2}y} & + & {P_{7}{xy}^{2}} & + & {P_{8}x^{3}} & + & {P_{9}y^{3}} & + & & \\{P_{10}x^{2}y^{2}} & + & {P_{11}x^{3}y} & + & {P_{12}{xy}^{3}} & + & {P_{13}x^{4}} & + & {P_{14}y^{4}} & + \\\ldots & & & & & & & & & \end{matrix}}$

According to one embodiment, one or more parameters in said sequence ofparameters, other than said subset of parameters, are set to respectivedefault values. In one example, the default values are set to 0.

According to one embodiment, said polynomial is a fully developedpolynomial.

According to one embodiment, said ordered sequence of terms is orderedby increasing degree of terms.

According to one embodiment, said ordered sequence of terms is furtherordered by an increasing maximum of x and y exponents.

According to one embodiment, said ordered sequence of terms is furtherordered by increasing x or y exponent.

According to one embodiment, a k-th parameter of said plurality ofparameters is associated with a k-th term of said ordered sequence ofterms.

According to one embodiment, said matrix coordinates are normalized.Normalization can be performed as

${x = {N\frac{x^{\prime}}{{size}_{x}}}},{y = {N\frac{y^{\prime}}{{size}_{y}}}},$

where x′ and y′ are the indices of the columns and rows of thequantization matrix, size_(x) is the horizontal size of the matrix,size_(y) is the vertical size of the matrix, and N is a normalizationvalue. In one example, N is set to 2.

According to one embodiment, said quantization matrix is used forquantizer step scaling or for quantizer parameter offset.

According to one embodiment, matrix coefficients are computed by aninteger process comprising limiting left-shifted polynomial terms to nbits by dropping most significant bits (including the sign bit), thenadding them together using adders also restricted to n-bit output (stillby dropping MSBs), and right-shifting the result by a value smax-sbaseto obtain a matrix coefficient, n being equal to matrix coefficientbitdepth (mbd) plus maximum allowed value of (smax-sbase).

According to one embodiment, when a matrix coefficient is furtherclipped to a minimum value, bit depth before clipping may be increasedby one to retain a sign bit, and in that case said n is increased byone.

According to one embodiment, said polynomial in (x,y) corresponds to onepolynomial in x and one polynomial in y.

According to one embodiment, variables x and y are replaced by u and v,with u=(x+y) and v=(x−y).

According to one embodiment, variables x and y are replaced by u and v,with u=(ax+by) and v=(bx−ay), with a+b=1.

According to one embodiment, two separable polynomials are combined togenerate a quantization matrix, for example M(x,y)=P₁(x)·P₂(y) or M(x,y)=P₁(x)+P₂(y).

An embodiment provides a computer program comprising instructions whichwhen executed by one or more processors cause the one or more processorsto perform the encoding method or decoding method according to any ofthe embodiments described above. One or more of the present embodimentsalso provide a computer readable storage medium having stored thereoninstructions for encoding or decoding video data according to themethods described above. One or more embodiments also provide a computerreadable storage medium having stored thereon a bitstream generatedaccording to the methods described above. One or more embodiments alsoprovide a method and apparatus for transmitting or receiving thebitstream generated according to the methods described above.

Various implementations involve decoding. “Decoding,” as used in thisapplication, can encompass all or part of the processes performed, forexample, on a received encoded sequence in order to produce a finaloutput suitable for display. In various embodiments, such processesinclude one or more of the processes typically performed by a decoder,for example, entropy decoding, inverse quantization, inversetransformation, and differential decoding. Whether the phrase “decodingprocess” is intended to refer specifically to a subset of operations orgenerally to the broader decoding process will be clear based on thecontext of the specific descriptions and is believed to be wellunderstood by those skilled in the art.

Various implementations involve encoding. In an analogous way to theabove discussion about “decoding”, “encoding” as used in thisapplication can encompass all or part of the processes performed, forexample, on an input video sequence in order to produce an encodedbitstream.

The implementations and aspects described herein can be implemented in,for example, a method or a process, an apparatus, a software program, adata stream, or a signal. Even if only discussed in the context of asingle form of implementation (for example, discussed only as a method),the implementation of features discussed can also be implemented inother forms (for example, an apparatus or program). An apparatus can beimplemented in, for example, appropriate hardware, software, andfirmware. The methods can be implemented in, for example, an apparatus,for example, a processor, which refers to processing devices in general,including, for example, a computer, a microprocessor, an integratedcircuit, or a programmable logic device. Processors also includecommunication devices, for example, computers, cell phones,portable/personal digital assistants (“PDAs”), and other devices thatfacilitate communication of information between end-users.

Reference to “one embodiment” or “an embodiment” or “one implementation”or “an implementation”, as well as other variations thereof, means thata particular feature, structure, characteristic, and so forth describedin connection with the embodiment is included in at least oneembodiment. Thus, the appearances of the phrase “in one embodiment” or“in an embodiment” or “in one implementation” or “in an implementation”,as well any other variations, appearing in various places throughoutthis application are not necessarily all referring to the sameembodiment.

Additionally, this application may refer to “determining” various piecesof information. Determining the information can include one or more of,for example, estimating the information, calculating the information,predicting the information, or retrieving the information from memory.

Further, this application may refer to “accessing” various pieces ofinformation. Accessing the information can include one or more of, forexample, receiving the information, retrieving the information (forexample, from memory), storing the information, moving the information,copying the information, calculating the information, determining theinformation, predicting the information, or estimating the information.

Additionally, this application may refer to “receiving” various piecesof information. Receiving is, as with “accessing”, intended to be abroad term. Receiving the information can include one or more of, forexample, accessing the information, or retrieving the information (forexample, from memory). Further, “receiving” is typically involved, inone way or another, during operations, for example, storing theinformation, processing the information, transmitting the information,moving the information, copying the information, erasing theinformation, calculating the information, determining the information,predicting the information, or estimating the information.

It is to be appreciated that the use of any of the following “/”,“and/or”, and “at least one of”, for example, in the cases of “A/B”, “Aand/or B” and “at least one of A and B”, is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of both options (A andB). As a further example, in the cases of “A, B, and/or C” and “at leastone of A, B, and C”, such phrasing is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of the third listedoption (C) only, or the selection of the first and the second listedoptions (A and B) only, or the selection of the first and third listedoptions (A and C) only, or the selection of the second and third listedoptions (B and C) only, or the selection of all three options (A and Band C). This may be extended, as is clear to one of ordinary skill inthis and related arts, for as many items as are listed.

As will be evident to one of ordinary skill in the art, implementationscan produce a variety of signals formatted to carry information that canbe, for example, stored or transmitted. The information can include, forexample, instructions for performing a method, or data produced by oneof the described implementations. For example, a signal can be formattedto carry the bitstream of a described embodiment. Such a signal can beformatted, for example, as an electromagnetic wave (for example, using aradio frequency portion of spectrum) or as a baseband signal. Theformatting can include, for example, encoding a data stream andmodulating a carrier with the encoded data stream. The information thatthe signal carries can be, for example, analog or digital information.The signal can be transmitted over a variety of different wired orwireless links, as is known. The signal can be stored on aprocessor-readable medium.

1. A method of video decoding, comprising: decoding at least a syntaxelement representing a value N, where N indicates a number of polynomialcoefficients, N being selected from among a plurality of values;obtaining N polynomial coefficients; forming a polynomial function withN monomials, wherein said polynomial function is a function of matrixcoordinates, and wherein each one of said N polynomial coefficients isassociated with a corresponding monomial of said N monomials; obtaininga quantization matrix based on said polynomial function of matrixcoordinates; and decoding a block of a picture based on saidquantization matrix.
 2. The method of claim 1, wherein a k-th polynomialcoefficient of said N polynomial coefficients is associated with a k-thmonomial of said N monomials.
 3. The method of claim 1, furthercomprising: decoding a set of polynomial coefficients, wherein said Npolynomial coefficients are determined responsive to said decoded set ofpolynomial coefficients.
 4. The method of claim 1, wherein saidmonomials are ordered by an increasing maximum of x and y exponents,where x and y indicate matrix coordinate.
 5. The method of claim 1,wherein said monomials are ordered by increasing x or y exponent, wherex and y indicate matrix coordinate.
 6. The method of claim 1, whereinsaid polynomial function is a function of a first variable and a secondvariable, wherein said first variable is a linear combination ofhorizontal coordinate and vertical coordinate, and wherein said secondvariable is a linear combination of horizontal coordinate and verticalcoordinate.
 7. An apparatus for video decoding, comprising: one or moreprocessors, wherein said one or more processors are configured to:decode at least a syntax element representing a value N, where Nindicates a number of polynomial coefficients, N being determined fromamong a plurality of values; obtain N polynomial coefficients; form apolynomial function with N monomials, wherein said polynomial functionis a function of matrix coordinates, and wherein each one of said Npolynomial coefficients is associated with a corresponding monomial ofsaid N monomials; obtain a quantization matrix based on said polynomialfunction of matrix coordinates; and decode a block of a picture based onsaid quantization matrix.
 8. The apparatus of claim 7, wherein a k-thpolynomial coefficient of said N polynomial coefficients is associatedwith a k-th monomial of said N monomials.
 9. The apparatus of claim 7,wherein said one or more processors are further configured to decode aset of polynomial coefficients, wherein said N polynomial coefficientsare determined responsive to said decoded set of polynomialcoefficients.
 10. The apparatus of claim 7, wherein said monomials areordered by an increasing maximum of x and y exponents, where x and yindicate matrix coordinate.
 11. The apparatus of claim 7, wherein saidmonomials are ordered by increasing x or y exponent, where x and yindicate matrix coordinate.
 12. The apparatus of claim 7, wherein saidpolynomial function is a function of a first variable and a secondvariable, wherein said first variable is a linear combination ofhorizontal coordinate and vertical coordinate, and wherein said secondvariable is a linear combination of horizontal coordinate and verticalcoordinate.
 13. The method of video encoding, comprising: determining avalue N, indicating a number of polynomial coefficients; encoding atleast a syntax element representing said value N; obtaining N polynomialcoefficients; forming a polynomial function with N monomials, whereinsaid polynomial function is a function of matrix coordinates, andwherein each one of said N polynomial coefficients is associated with acorresponding monomial of said N monomials; obtaining a quantizationmatrix based on said polynomial function of matrix coordinates; andencoding a block of a picture based on said quantization matrix.
 14. Themethod of claim 13, wherein a k-th polynomial coefficient of said Npolynomial coefficients is associated with a k-th monomial of said Nmonomials.
 15. The method of claim 13, further comprising: encoding aset of polynomial coefficients, wherein said N polynomial coefficientsare determined responsive to said set of polynomial coefficients. 16.The method of claim 13, wherein said polynomial function is a functionof a to first variable and a second variable, wherein said firstvariable is a linear combination of horizontal coordinate and verticalcoordinate, and wherein said second variable is a linear combination ofhorizontal coordinate and vertical coordinate.
 17. An apparatus forvideo encoding, comprising: one or more processors, wherein said one ormore processors are configured to: determine a value N, indicating anumber of polynomial coefficients; encode at least a syntax elementrepresenting said value N; obtain N polynomial coefficients; form apolynomial function with N monomials, wherein said polynomial functionis a function of matrix coordinates, and wherein each one of said Npolynomial coefficients is associated with a corresponding monomial ofsaid N monomials; obtain a quantization matrix based on said polynomialfunction of matrix coordinates; and encode a block of a picture based onsaid quantization matrix.
 18. The apparatus of claim 17, wherein a k-thpolynomial coefficient of said N polynomial coefficients is associatedwith a k-th monomial of said N monomials.
 19. The apparatus of claim 17,said one or more processors are further configured to encode a set ofpolynomial coefficients, wherein said N polynomial coefficients aredetermined responsive to said set of polynomial coefficients.
 20. Theapparatus of claim 17, wherein said polynomial function is a function ofa first variable and a second variable, wherein said first variable is alinear combination of horizontal coordinate and vertical coordinate, andwherein said second variable is a linear combination of horizontalcoordinate and vertical coordinate.